{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 11.4 高维向量可视化\n",
    "在第6章迁移学习一节介绍中过，在ImageNet上训练好的卷积神经网络模型的卷积层可以被看成是对图片进行特征提取的过程，那么这个特征提取结果的效果在没有可视化的情况下是不容易被直观判断的。类似地，第9章中介绍的单词向量也很难直观地了解其效果。\n",
    "\n",
    "**为了更加直观的让大家了解这类embedding向量的效果， TensorBoard提供了PROJECTOR界面来可视化高维向量之间的关系。**PROJECTOR界面可以非常方便地可视化多个高维向量之间的关系。比如在图像迁移学习中可以将一组目标问题的图片通过训练好的卷积层得到瓶颈层，这些瓶颈层向量就是多个高维向量。如果在目标问题图像数据集上同一种类的图片在经过卷积层之后得到的瓶颈层向量在空间中比较接近，那么这样迁移学习得到的结果就有可能会更好。类似地，在训练单词向量时，如果语义相近的单词所对应的向量在空间中的距离也比较接近的话，那么自然语言模型的效果也有可能会更好。\n",
    "\n",
    "为了更直观地介绍TensorBoard PROJECTOR的使用方法，本节将给出一个MNIST的样例程序。这个样例程序在MNIST数据上训练了一个简单的全连接神经网络。本节将展示在训练100轮和10000轮之后，测试数据经过整个神经网络得到的输出层向量通过PROJECTOR得到的可视化结果。**为了在PROJECTOR中更好地展示MNIST图片信息以及每张图片对应的真实标签，PROJECTOR要求用户准备一个sprite图像和一个tsv文件给出每张图片对应的标签信**。以下代码给出了如何使用MNIST测试数据生成PROJECTOR所需要的这两个文件。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From <ipython-input-1-168a9ebd240f>:40: read_data_sets (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use alternatives such as official/mnist/dataset.py from tensorflow/models.\n",
      "WARNING:tensorflow:From d:\\python3\\tfgpu\\dl+\\lib\\site-packages\\tensorflow\\contrib\\learn\\python\\learn\\datasets\\mnist.py:260: maybe_download (from tensorflow.contrib.learn.python.learn.datasets.base) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please write your own downloading logic.\n",
      "WARNING:tensorflow:From d:\\python3\\tfgpu\\dl+\\lib\\site-packages\\tensorflow\\contrib\\learn\\python\\learn\\datasets\\mnist.py:262: extract_images (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use tf.data to implement this functionality.\n",
      "Extracting ../../datasets/MNIST_data\\train-images-idx3-ubyte.gz\n",
      "WARNING:tensorflow:From d:\\python3\\tfgpu\\dl+\\lib\\site-packages\\tensorflow\\contrib\\learn\\python\\learn\\datasets\\mnist.py:267: extract_labels (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use tf.data to implement this functionality.\n",
      "Extracting ../../datasets/MNIST_data\\train-labels-idx1-ubyte.gz\n",
      "Extracting ../../datasets/MNIST_data\\t10k-images-idx3-ubyte.gz\n",
      "Extracting ../../datasets/MNIST_data\\t10k-labels-idx1-ubyte.gz\n",
      "WARNING:tensorflow:From d:\\python3\\tfgpu\\dl+\\lib\\site-packages\\tensorflow\\contrib\\learn\\python\\learn\\datasets\\mnist.py:290: DataSet.__init__ (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use alternatives such as official/mnist/dataset.py from tensorflow/models.\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAQsAAAD8CAYAAABgtYFHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvFvnyVgAAIABJREFUeJzsXXvcVVP+fvY5b0gylUKiYTCamJGK3EuDhMi9ZtwvJeM6UvkJqaGLa7mkMiSG5JYYVIqYEVJJRrrp4hLRRSFd9P398b7P3mt/91rr7HN6uzjO8/m8n/fsvddee+21117ru9d6nu83EBGUUEIJJeRCZnMXoIQSSvhloNRZlFBCCalQ6ixKKKGEVCh1FiWUUEIqlDqLEkooIRVKnUUJJZSQCpu8swiC4LggCGYGQTAnCIJum/r6JZRQQmEINiXPIgiCLIBZAI4B8DmASQDai8jHm6wQJZRQQkHY1JbFQQDmiMinIrIGwHAAJ2/iMpRQQgkFoGwTX68egM+M7c8BNDMTBEHQAUAHAKhWrVqTBg0aAAAmT56MJk2ahOk++eQTNGjQAF999RV23nln70U/++wz7LbbbrF9q1atQtWqVRNp9XWmTp2KAw44IM29eaHzzSdtrm0AYT38+OOP2HbbbTe4vLnKRHzxxReoV69e4vjatWuxevVqbLfdduG+OXPmYK+99sLChQtRv35963VmzJiBP/zhD7F98+bNwx577BFuf/zxx2jYsCHWrFmDrbbaqqC6nTZtGvbff//E8VWrVuGLL77AXnvtFaadP38+dt999zDNTz/9hG222SZ23qJFi1C3bt1EfnwetjIuWbIEO+ywg7OMNkyZMgWNGzdO1SZWrlyJ6tWrh9vr1q1DWVnylZ88efK3IlLHekETIrLJ/gCcAeAhY/scAPe60jdp0kSINWvWiIjIuHHjRESkXr16khZbbbWViIi8++67IiJy8MEHyzvvvCMiItWqVRMRkcmTJzvP33fffWPbH374YeprL1y4UEREgiCI7c9kMs5zbrjhBhER+eyzz0REZPHixdZ0QRA4823Xrl3qMh5zzDGx7d/+9rfWa5n5b7vtts78WM8uVKlSJXx+vMdsNpuqrGa9vf322yIist9++yWOabzyyisiIlLe5EWuv/762PF99tlH/va3v0mrVq0S515wwQWpyiYi0r59+9g26y0IAvnuu++s59x9990iYi//VVddFdv++eefRUSkbdu2IiKy1157iYjI0qVLE+fynWE9Mf9atWrF0gF4X9K8v2kSVdYfgEMAjDa2rwdwvSu92VnkgyVLlhR03saE7WFuDFSvXl1ERL799tucaX0v18Y4984770zsY8eo8fHHHzvzadGiRd7Xzge6A27UqFHqc3/zm9/EtufPn+9Me8opp3ivW1n48ccfvcfTdhabes5iEoC9gyDYIwiCrQC0AzCqkIwymfKiX3LJJQCA5cuXh8f0Jwdx5plnOvMbM2YMACCbzQJAaK5x23Zt/j/11FMBAOvXr4/tN1GzZs1YfjvuuGN4rHfv3tYypSlL586dY9vVqlUDgIR5e9hhhyXOHz16dGzbVo799tsvtj1kyBBrWU188803AIAnnngCAHDEEUcAAP7+978n0v7jH/8AAHz++eex/fwU0XW5atWqRH0wzccfu+fJbc8EAK655prEPj5HYubMmc58NVasWAEgemZ77rln7HiPHj0QBAEAhJ9B7du3B4BwP1FWVpZ4ZpdffjkAe1sAytsxj/FTj59QzP/cc89NfT8mNulqCAAEQXA8gHsAZAE8LCK3utI2bdpU3n///U1WthJK+DUiCILJItI0V7pNzrMQkZdF5PcisqevoyD06GGboNFYvHixdT/P7dmzZ+IYJ0n1aH7ppZfmvN6HH34Y227evHn4m/nx/xtvvJEzP8KcnMqFcePGeY+XlZXhsccei+1jnQ4dOhQAMHz4cADAyJEjnSOXPteGPn36JK4NAD///LM3TyA5uvbo0cOZVpfRZwW6tnmOzerZeuutnfkSHMyOOuqonGXReP7552PbfN5VqlRxnsN833nnndj+vffeO/ytLV8N06rNB5vcssgHJcuiBB/Wr1/v7bRKSIct1rIoFHqE1vttFsfxxx8f23766acT2/q86dOn5yzLypUrAUTftr65EOL777+PnXP++efHjgdBkBhVDzzwQAD+UZygVeBLq60aPbdDi8y2pKjhWvr0wWbxXXzxxQCiuZDrrrsOAPDAAw/E0nXs2DE2V9GpUyc89NBD4baJ2rVrh7+11Ue46umRRx7BokWLALjnQHwjvws+C+n000/3lqnQ6wwbNgxA1I7+8pe/xNJeeOGFeeX9i7EsChlFRCTxAv7a0K1bt8RnQQkbD8OHD0e7du02dzHyQtFYFhw16tQp54x061YuJ/npp58AAOedd14sHYDwYbGjWLZsmTVPE1dffbX1mK2D4uoH8dln5TwzEpAOPPBAZDKZ2Ojo6uhsZeG+GTNmWM8170d/n/bq1QsAMGDAgJzXMYlOAHDHHXck0rAOdVpaJcyX8x4m9Hc8QevBhqVLlzqPEb5RGojPe9SuXRtt27YNVxxcmD9/fvh78uTJsWPaIvWBFh7z+/HHHxNp+Kxc8xqcP+PKVj54/fXXw3wnTpwIoJwgZyLNfIoVadZXN9cfKgg0IiKtW7cWEZGOHTuKSLTOb673k9jD9Wqen4YTkE/aDckjLflIJFqH5zr/k08+GTt+8MEHJ85p2LChiIj0798/Z1l4jGW65ZZbcpbVRQLbcccdE2nN5yciss022zjLMnTo0Fj+t912mzMtQb4Fy8n/vK5JZlu3bl3s3GHDholIkqBkw6GHHmrd//rrr4dlGDx4sIiIfPrppyIicvPNN+fMlyiEX7HLLruEZRCJ+DU+vPjiiyKSbBPYEklZ+f6Zjc3XKIn169eLSET+4UPYfffdY9s333xzWMlTpkwREZF77703lobwkYNWrFjhPObCf//7XxFJMhczmYxkMhm55pprEtfu0qVL3texdRLXXntt7Nj5558fO062KVG7dm2ZNm1abN+yZctEJGqk5vVWrlzpLcuXX34pjzzySOzYTjvtJJdeeqkMGTJERETKyspi59juo0ePHiISsRjJsGTavn37Js557bXXYtuakWqibt261vtgp80ymtD5cFu3J1v75TNv2rSpiIgcffTRImIfDFh/PMc3GEydOjW2ff/994uIyEUXXSQiUfsqis6iSZMmsueee4pINKo+/fTTIiKy8847OyuJ6NatW840BGmz2iohfA/l/fffj51bCNhZiIgMHDhQREQWLFgQS6NHer5gtvJxe/r06bGymenYQRJmR6VRs2bN2Laun2XLljnrqFOnTiKSZDfayqtfxMaNG8e2TTbiF198ETvGe7WBlP077rjDmUZjn332EZGktMDVKZrgs+JARYvPfIbz5s0TEZGuXbvGzv3rX/8a/tYWFuuSsDFEH3vsMRER+emnn0Qk2bFrFE1nQXTo0MF7wyaeeOKJ2LY2U3147rnnRCTSk/jAl+vNN98UEXtncc4554hI1DC6d+8uIhFfvxDMnDnTeey4447LO79zzz03tm2rJ+7jpxG3baOfpjFzNOU5s2bNcpbFRgnPhdtvvz227evYWd7atWuH+3z1KSKy/fbbW/f7rNxcWLRoUfibn9bUuPiQz2fy2WefLSIiderU8aYrms5CN342uMsvv9x58y7BDCvNfBn42yaeEvFbCxx5qlSp4kyjy6SvS5jiK+bnOof7aWWZ5dTnBEEgRx11lLVM+t5uvPHG2DZFe+a12YG7RFG28prlpmAvCAJp1KiRvPnmm3LFFVfkzMv2HPScy+9///twP+e4NLRFxLkdPksTzINtUAsIgyBIfC41b97cWV5X+V0dAD/5fOdSEOnDoEGDvMeLprMoFtCUveeeezZzSUrYUuFSF28oXAOr8cm9RQrJCsKXX34Z/uayqLnUNXv27Fj6evXqAUhSh21EIlJs9XKST5REaKKYSfDKZrOoV69eWIZ8lt8IkmkI0nSbNm2aKO+TTz4JIFr+NMui68FX7gcffDCW1kWXZv3oMpro378/gIhw5Vu+vOyyy2L5p1ne0+SoWrVqoW3btrEym+0EAN59910AwODBgwEA22+/PQA7DX/XXXcFEC3Vsh5tBECXDIFl2WeffRLH9D2SHmCj+WuhXdu2ba3X69ixY6IO77//fgBAhw4dYmm1YC4XfjGkrBJKKGHjoGhIWSSWsFe3kZtcoxDprV9//XXO6+y0006xbVOYozFv3rzYNglLS5YsCfdxJDnssMMARESuCRMmWPM84ogjwtGVVsi+++7rLbPtvnkfHKWIRo0aYdSouDcAV72lkTAXTOypgC4L4RI5ZbNZ/OY3v7GWgW4KaGlQEu+Dlrlns9mEdaAJaY8//jgAoEWLFuE59JjFdrlu3TprGSsbPmbyxmItlyyLEjY6zjnnnITitYQtB0VjWWhw1D3uuOPCfbr35rem9uVIRyfdu3fHnDlzALi/kW1S5mw2i9q1aydGIJ/sXDu9IdhJ+0Ye5mtLc8wxx1jPof9NXf633norzOvZZ5+1nqtHRRP5iKd0/bCjmDZtmrVsJrRLAtsouWrVqtgxzo0QN954Y/hbU/Nfe+0163W32morZ1kWLFgQ29+yZUtn+TUoIDTB8mvQcc4HH3zgzM8lqPShUaNGAKJnWFZWht133916z16kmQXdXH+21ZAvv/wy7SRwiAYNGuR9ThqQPOWCyfegP0QiH8JYGviWknNhp512Sp2WS3ZvvPFGzrTffPNNbPvZZ5/Nr2AbEVxOLoTer6njaZBmqTkf6Db91ltvFZwXimXplBRVyw06b55rz1orQphsSQ3NMnzooYdi56XByy+/HP4mI9FVXt9SKst/2mmniUjc+SsbeyFUcNd1mL/JASFJjRToOXPmiEjEBTn22GNz5s968/mCJNWZYH2Z/BHWZf369UUkYke6+Ch0xmxLozkaP/zwQ3iM1yTp7oEHHoidaxt8vvrqK+t9+Z6vLtOYMWNExM9EJalM+wU12beat0IdyejRo0UkeoZE0XQWhCYk+ToLglqIW2+9VUTSeeWmVkTDFEFRJMQyaOe4jz76aPh77dq11vLyPmz6Et6rSbqywewENS04nxHTpWMQEXnwwQdjZdp1111FJMkU5ctiiq6Y3wsvvJCzTCTFUZ9ie74DBgyIlcWFbDYb67BFIg/aJnErV5l0B0b4znFZXLZzWD/6xfd1Fjo/1oWN2MYOnfehny/lAkXXWdCMYyM68sgjfXUpIpG7d+pIbFRi7iO7k70wHwZdr5vgsSuvvFJERK6++moR8YvOCGoENE3a5m6eDzeN9cAy/e53v8uZVoNaAt9LSB0JXzbi5JNPdp4zY8YM637fy6Zd9G8s/Pvf/w7LMmLEiNgxWm25WJjmcf0pp89lZyUSDRCFfALpTpTqWRMXX3xxLH92+AzfoOn4RddZ5FOxhx9+eOq0RD5xNjRWrVolIpGi1HZ9lv+uu+7KmZ/rpaWZaovZoS0u3VjZYExASeptoqRceoV//etfse1nn302p1Tftp8vG3U2vrQ6f99L/cwzz8S2Tz/99LCcInYFqa5/XwfmUh6zTFqAZ8JVP/y8oqo2n3NNVK1aVUSiAdHmIqBJkybF11lo0CT1wdYQrrvuOmvj6tmzZ878NPgwbJLotPApPTW+/vpr5zEGx6G8mRNey5cvL7hsJs4444zYNi27Sy+9NJGWlgpx5plnVkoZih2ueQ8T5hyaSFTXtjZNazmXhVR0ncWrr75qvVFfD8tK6tOnj4hElad1/iLJz498LBnfTDe/14mDDjpIRKKOhjAnctmB0G+DhjmaUezE+RkN27csI5DRhN1hhx1ERKRGjRoiYpc0u+qDczJp4KvTfv36pc5Hj/y+zycqSvfff38Riep26623TqRle5k7d66IiPzpT38SkaS1c9hhhyXOZeQwrbAl2JnPnj3bGc1N1485L6Sl6QQ76ypVqkifPn1kwoQJ0rlzZ2taF9J2FiVSVpFjt912C93+lZAfRo4c6dRgFBOKgpS1dOnS0K8lacCaptuvX7/EeSRlpcGdd94Zy1d7BGcEKBt4Dn0cDho0KJGGhKF8yDRanMVzDj744HBfv379kM1mQ0o7I5Ax7TPPPAMgLsLT5dY4+uijvceBJAHKh6+++goA8N1336U+h9DPwYaTTjrJej0bKC7UOOecc5znfPvttwCAV155JbafUcbKysqcdTVixIjEvr59+1rTano6yWYkAtrg2n/33XeHojP6/2TaVq1axdIOHjw4L3JXybIooYRfOYrCsvCB0uOFCxfmTEtZsg06doKmG5vUZG0dUPhF0Mpp1qwZgHIKse65jz32WACRR3CTfnzLLbcAiIRKpLQXEqeCIx7Lv8suu+Q85//+7/9i27aRzTUSHX744eF1KbhijNCxY8cCQBiPo3r16li9enXsfFKPXZ67SWMfP3584tjcuXMBIKTwm+D9a4FagwYNAADPPfccADcF28yf9969e3dnWpcFacoT0uKaa65JWFj0ak/Qu71p4eiYNIzju8FIM7Gxuf62VOc3nIjaENd4afIvFiAFgW5DkM+KUrHCJz2gL04yaAv17v2Lsyx0j63jWQBJ8ZH+5qxbty6A8lHnxBNPBBB946eJ1cCYnRxNaRGUlZWhZcuWqYIhMbK1rbw6JiiPv/jiiwDi8vN8RE0arujg+jv8kEMOCeudkdHTgPFDpeJT9+yzzwZgt0606I+WxyGHHBLuoxVAMB89j6Lrv27dugmLUac165z7atSoYb2eHt0rG7TMCNu8nIYZk5fWGa0SxmytWrUqgGiuJ+8IaGl6lM31Z7MsyJ3gMlcuZ6si6Rz1srd96qmnRCS/0VBrBUydAUGnuORKuLyI28A0XP7zweVv0wabFiQXSGf2kY00SL32rfdrRiud5FLPYIL7KM6jYO+ss85y5p+GwKVhY+/mQhqqtgvaazgd+dpQCGNXSyYIFAvPQjdk8gr0fgZQMSuDa9saPuev9Pps+xSgKKh3797Wc2kK2jo5lle7r7eB3AWWk52RpiWbWLp0aewcG8gzYYiBJUuWiEjk2Nj3+TN27NjYNj1qjx8/XkTi3tB1MCSCZbOR5Yh8OmlCvwTmfWgeyCuvvCIiIn/7299i55psSV0GsnttnWquT0beMztMMxgQPb8TaeQCVF2PHDkyZ1rdQery33DDDSKyiToLAPMBTAfwAS8IoBaAsQBmV/yvWbE/ADAAwBwAHwJonCt/s7OgDmO77baL3XD16tVDtqIeWeitevbs2bHK69y5c9hgyUQkDdgGUprbt28f2+9SO9rCFlDpR8Ukwfu58sorQ1IUR20XKcsnb2dD57kkXLFs5sujmZa8vonVq1eLSLITcjFe77333jDWi6v8IklqOYlzFOnpYEQmSCrTZSLt3od8rChCtyte1+z42Xm6lM587tRt2PKj634ffMGRXKBimGXbb7/9woBcIpu2s6it9vUD0K3idzcAfSt+Hw/glYpO42AA7+bKv5AJTjYkG2wqQlY6r5VmZNMmM7n8PhQSgMjo+UWksIZOXQMjoIlErE4dwmBDgiTZztWBm2xg/dMy4TYZtWmgpept2rQJj+kX0DXK0tKwpdXXsUG/vDq/QiwmHx5++GHnsZNOOklEokGH4Rd22203ESmXPKiybbbOYiaAuhW/6wKYWfF7EID2tnSuv+233z6kHqeJzUFwlErzcvXr109Wr15dkDaE38400bWfAJFoToWmvw5buCFo2rRpaJYyP8qOTQGVlkDr0c/3MtM8N0ciEb9vCsKlOg2CQIIgsCom6Vjmn//8p4ik64jTIB/nQGlH7QYNGoRzRBT5ESx/mryYRkdfM0FtDtNqPyKcvzGvl7bzT9tZbBApKwiCeQCWARAAg0RkcBAEy0WkhpFmmYjUDILgJQB9ROQ/FfvHAegqIu+rPDsA6AAA9evXb6Jdmm0J+PHHH7Htttti7dq1BXEgfu2YM2cO9tprL2+adevWeZmkJVQeNhUp6zARaQygNYC/BUFwpK9Mln2JnkpEBotIUxFpunDhQgRBgAMPPDBsON26dcu7kDoWBlC+dDl16lQr+cqErcGSIKQ7ChtF2UVqOv30053l5TVJJmPZrrjiCuc5GnopMg24bGyLccEyrVy50nqubT/P4bJk8+bNAZQTonS9clmU9cPlvptvvtlZXnMJ3IS5bXpcB4ApU6YAiC9PHn300fjHP/4BIKJIm2VJA1L0d95555xpn3rqKev+P/7xj85zuOzJMmlv9Bo2Qp25RM9nkQ8qje4dBEEPAN8DuARACxFZFARBXQBviMg+QRAMqvj9ZEX6mUznyrNE9y6hhI2PjW5ZBEFQLQiC6vwN4FgAHwEYBeC8imTnAXih4vcoAOcG5TgYwHe+joJ47733YtscAegJ2aTx2khLNjzwwAPOY4ya5evlieHDh8e2KTzq2bNnYrRzjTi1a9d25u8bMdOCXr0XL16c97lVq1YNhWj0/O2KSWFGtxowYAAA4KyzzgIALFu2DADQp08fAHZrjSMl/48bNw5AuTBKgxaWJq/ZoGONaOo/CXXmKPzmm28CiERmtjIA5dbJyy+/jNatWzufzfTp03OWkVaOaZmmtWpc8W26dOkS/v7oo49iebrIeLlQsGURBMHvADxfsVkG4AkRuTUIgh0AjABQH8BCAGeIyNKgvJXdB+A4AD8CuEDPV2j8UiyLkgz814Fq1arhhx9+2NzFqHRsdMtCRD4Vkf0r/vYVkVsr9i8RkT+LyN4V/5dW7BcR+ZuI7Ckif8zVUeQCe0kzfqOOATJ69OjYfh+YhpJvG2bMmBHbpvhJd2hmtC0dM7Vr164AIovIJt9mWXyCJaCc5uyKd+KjgdeqVSu2TUGcGW+DMVb+85//eMtgsxJYJtK9CcbwNAVrbdq0iZWbo6AtT+Z7wAEHAAAuuOACABGNXF/fBlqdWmhng56ToiSASBN3wyZH0GB8WiKbzWL69OnW+9AuGnzwteWWLVt6JfZWpFky2Vx/Ph+cZFqacDE24VnjTkvPtbmPI8477zwRiZazbMtlXDrNR3xGshmZft27dxeRdEtiZCrmA+3gVUTkz3/+cyyNrd7zhQ63YIP2MJYGd9xxRyHFSYAcDy5/apicFRcKiYieZpmV3s+nTZuWM6328OUCioXuTWhlISuWNNpMJuN0nkpMnDgx/E3+BjUbzI+0cfIL0jxANtKhQ4c602hego3a67oW3bCxkxg0aJCIxLkaZAgyLIGv3HQLR3Ym05LxaoK8DYKhADRle/r06VZ3c7nAOiMbUusjMplMYh+v3atXLxFJOtS13Tv3pQnGo89nPWnk4wrQ1sFzn86fupjzzz8/3Ofi5dBLuUi83L169QoHsVycnqLpLEj/zVVpIhFjkNCxFUww30mTJlnPcY0qJmht8Jw0UadcTnfNMurGuiHMSpKcmH+rVq3kvffei6XxufPXYNl8TnjpA5IWHanb+VCUSYvWhDJbWXxkJo1CyHDU/Nh8Z2q3+mQJ63s1SYUuDUia58x2q2nfvrrVGpGiFZLNnj1bBgwYkNcLRJ0HAwalaaSuyl+/fr0sW7YsllY3OFdZbHT1NA2CrE/+50undTEmRo0aFdum+Kx58+Y5r0ekqSdaFgR9JYjkDorEUTCfTzFbqEA6xyVY7nxYvtRL+FBIJ63p5Lfccoszba761jFaTLDTYLwZW160nnnMxWItis7CJr/lw/C5QCe08Eu/UCK56cQ333xz7EGYvxm8SJeBI/ewYcPCl5Z0aV7vxBNPFBGRffbZx3t9E7y22QC1GEwjm806Q0AyPyoheT8+aAl8mpGa9eOTkOt5JSphzTxcEcLSXFsPAjpIlfkJa4Y9NM/ZEPjaKetQi+uqVauWOoxCpiIk57333puwxli3VAibyteK47/8zsInJGN8jGbNmvnqcKPCN3FaGdCSdFujzeXPY2N53co1P2QDy6I/F03YVJkbE2aoycrAvHnzKjU/Dc7d2QJCaTBGLkNDulAUnYXZQzKokI6qNGvWLKd/BD3qcZbbVI0yzQknnBBL64vEpV9Am1Rdi7X0yMaOhrPbJvSLyIee67oikeLQTMN0eiKNI0yaTu+DDz4QkcgHhgu1atWSBQsWyF//+tdwH7/RM5mM9O7dO/SrYQNXEcyYqSLl0nqbUE8kckHgg8vZEB3cmP44CB1fxjU/kMZJDicbTTAWDv2j6OhpJmzBvTV0LBqz3kXcnzVpO4uSd+8SSthCEAQBNsf7WBTevdesWRPG5CDNmIQc4tVXXw1/a6LK7bffDgCYNWtWbP/kyZND/5ku8HqatDJjxowEhddH7KHAiGl4PwQp0SZ4H6S6M17I999/7y2ziXbt2uVM8/DDD8euR+g6BoBOnToBAD755BMAUewMGzmodevWACIv5RomscgltNMxQUghN9PwP2nfNs/ahx12mLUMBOvARQYzy0tq9cqVK7F48WKr53eCgjR6oTfBfVrgyPvgub179wZQ3nZ4LRe9m8/HBKULJIYxD1LgXdR9F0qWRQkl/MpRFJaFCfaKLnpqNpsNZc3UaVB0lk/+++67b2y/OQoy/oLLwzVHhkwmE1o1rutwJKpZsyYAoGnTpqHVkYZ6rsH4EdrKodjNHPnWrFkT21eIQI1IQxfWzyENVZllssUCIY08n3Lra9avXz+2bcr/XeVj3BPipZdeCn8zGhrbDyX3EyZMiJXZxNtvv20tI6OZTZw4MbS+hg4dCiDpcZxtPZvNxkSXr776apgfLRmTzl8Q0kxsbK6/LTVuSC6kiYbtg807+JYG+iW95JJLNnNJ8vM0Xgg4QZjPMjdBT2Yme1ijEG9gnPD3IQ1bVST9BOcWb1lss8023uPLly8HANx2222JYy+88EJiH7Hbbrt582VAXNvoRUckvDbnEigOo8zadKRCpHGyw29MHT/CFtOEIxdHMkbaIjjfkc1mceqpp1qvnatsNjB2Bkfodu3ahSNZr169rOdQSGbD88+XC5hnz54d289Rtnfv3k5lryuW6n333YdJkybF8tGgRWZleFl0AAAgAElEQVRapsSjjz4KIJoX43yNDRQIatBBjzl3ceCBB6JOnTqhIx5G1fO5TiDycYSk5+XoCMiMxZIX0vQom+uvSZMmVgZfvkjDxOPyJKFjgVx99dWJc+gpe8cdd8y7LGlGZDiWNDt16pTQmnBpk3AxN33Llhpac2GCBB+OujbCFUV6ZBv6LADqSkgVJ1x1YCINl8QlGPz0009FRGTrrbdOHCOp7/DDD8+Zf5cuXaz7//vf/4qISMOGDXPmsdNOO4mIn+hGPUwhtHVqTbTlimLgWaThu/uwaNEi6342QDMPUoX5ny81X0JTaZhW8EU88sgjznNs6kQKp1y8AhP8VNP5k0PBBt+/f//wGNf1NeXZF0+F/995551YWrrfX7FiReqgPL6GTp4IaeUMZWBCv9jkFzC+h9ZwmApc3oev4yKd3BXqgeJDEZGHHnpIRNwDUj5t2BVvxUzres5poDk+RFF0FuacBR9GixYtrJXmq5w00PoCn4Dp+eefF5GIe68D2ZjldYHHJ0yYkDjGgEEE5wc4CvrActPyMN2+a2uJ0J6pbdAvuBYn2fDNN9/Etmmp2EZh7Wn8T3/6k4gkAzqZIDORrE8dTsBEvXr1Ysf4XxOZTJGdzocUfSpk8+kIfLoZajbSiMJcA5INtCRclHG2laLpLCwKORHx+wugC3rdEFyWhkgUX4JiLT5AbRaLVE58DcaV8DUMnwk+a9as2HYaM5dw+YrgZJwPNKt90BoO3nMh1OrjjjvOeYy0cX7CaIWyDXPnzhURt+zcxIY8Z0LL602wfRZyHd1526w1Dhg+AaJIEXUW9BXhMqFMMPDKu+++az1Ok/aoo46SxYsXy6uvvhpSxdNaIY899lj4cLUknS+bqcQkXPlnMhmrOtWEzwr53//+F9vWkmkT999/vyxZsiS1atYEz2G8Va1oTJOf7uBM8FNBK3xN+T+FgNraoPIyzfwGQco/y8q4GyJxPxJmGQhf+EXeY64XVKSwuLT0W0I/IJx/OO2006Rz584xnyT6OZifoiLlFke7du2Kp7PYEtGpUycRiT6JiHyC1G4JGD58+Ga7tg5aVMLmQ1F0FgDCEY3Ru3UvbwZEJvhtzu/UyjDzbGBZ9AiRz1yJWbZ8PiUIrv3zmgzdx1FWBw3+8ccfraH6XOCkWxohk4ZrpNeiPZGk+z5+FupQkTZwFcEm1iK4esPVBNtqUb169QTlsWxEJIpfSmEfwz4yLzOKHb2X0csVkU9b0J+BZ555Ztju88nPFS1dh3LMZrOSyWRKQrJfK8466yxnEJsSSrChaOjeJOKk8cZM6DT0VK1hCmkY84MgOei4445LpKU3bMJF7c5ms2jSpEn4GygXsQFAvXr1/DeBiOxl3s8111yDbDaLGjVq4JZbbgnJZUzjinFhoxv76N7Dhg2LCb1IamI0Nu4nqckGTcJy0ajN61D0N3LkSGe+Grw30pknTpzoTJtPlDGCbYD1xP9XX311IvIbCXu6Thl3BYjEbWzTJOGxjulV/Mknn3SWSQvKVq9e7UzLstx3333ONGnwq7IsRCRvpV0JmwZNmzZNhFQoYdOgaCwLjRNPPBEA8Omnn+ZMS9kxe3DdUaxcuTLcR5EYwf2MpmWCo9O9994b2//1118DAAYOHIjJkydj3bp1Ya/eo0eP2Lk+6NigpI/baLq8t1WrVnnz/Oabb0Lr6YMPPshZBuathWK8jr6eLeKaKxbpdtttl0jLjqJ///6x/bvuuqszXz4jXac+WXouy7RBgwapYn2YmDBhAnbZZZfYPh1jZtiwYWjVqlVsn6st6DLefffdYTS366+/PnW5dthhBwDAxRdfHLsen6mLou5EmomNzfXXpEmTBLFHw0dU4oRRnz59vHlsKAqh3uolzy0ddLZbSLk5qZgGhbgBzMcVHydqSeG2cT/0KpxtKbxQbM7nrpd8Bw8eLCJFshrSpEmTBPkqjWs57uPsL9emzRl9VyfEtMSee+6ZSEMuAIPx3HTTTTnLxt+kcNM9na/8LkbftddeKyIizz77bLiP/AcG8PGtAHF2Px+dCIlu2vcjg920bt064d6O5T7ooINy5p+rkzA7ZAZzcrm21533unXrcua/1157eQlUzEcjDZPVVkZzHyn59HquPaiLlK/wZTIZpzNh8msAJIIg6frQfJ2i6SzIadDgwzEti7p168rkyZMTaVu3bh3bLoQGbru2TkO/k1OmTAk1Axpc5iOl+4EHHnBec7fddhORdPx/7TM0l5NWEZF99903ts2gOdqztgntg5OMSB/I+szHArvooosS+0i31rj11ltj+ev6Ml/yHj16iEj0DGk12F52RoDLdX2RiCjmWrasLPgIYebAZhMBuspUNJ0FodebObrYkE+jZFp2KDq0Ho9zzd0G/RCmTJmS2Ef+A1mfNrGZq2zEE088ISJ2yrWLDegLlkQfC+QyMEyiDywTO700vh7SaB5csL3EOjodGZy+F4nlZhpNSWceNmiFsM+ZM5GGH0LQca8POkwD74c+K7LZrKxYsSK0OtOAbaYoOouaNWsmoorpEIGaBCOSjOfB/6ycK6+8MnGO7ox84Qu5b9iwYSKSJPgwhKBIJEzS+dxwww0iEo1Ib7zxRugERcc74XyB72Wj4lO72bd1nPQo7qIx2+YYtMdx/VxMaD2Hq9y+KGzaO7oN/IzS+h3tMf3II49M0NW1526zLPwc0SSpNIMQn6v+FOB9TJ8+PdY+TNg8jIvYP2d92hD+5v977rnHmZ9IkXQWNrp3IXEZ0vgjIGim2sxgF3wxTjXYwfhgutHPF/x02RBo3xgm0nzebAhMVuSWgrFjx26S66CCOfr+++8njuVjqWjocJWW61ZOZwHgYQCLAXxk7KsFYCyA2RX/a1bsDwAMADAHwIcAGhvnnFeRfjaA89IUztZZcPRjT51rUsoGHWvSV5mm0xtGaWfwH5fVQFx//fU59SK2iNwUvJH+7Zq3sYFWFTyiKu0jguUnjVnvt8HmC0SrWfmM6BzIFwuWylEtdnJNYopEnwO0Cinlr2zcdtttOdNo5z+6/qlyzWaz4XxWrmjyGSPmC6EtLZtlTejJaJeVlrazyEnKCoLgSADfAxgmIvtV7OsHYKmI9AmCoFtFZ9E1CILjAVwB4HgAzQD0F5FmQRDUAvA+gKYo595PBtBERJIkBgMluncJJWx8VBopS0TeBLBU7T4ZwKMVvx8F0NbYTzv7HQA1giCoC6AVgLEisrSigxgL4LiUNwIgIjy54kzEbkrRcgmbD8sPP/zQe10Nk5pM0heptiRerVixwlkmXV5bfA+SZjQhSnuX/uMf/xgrlw9vvPEGAGD9+vXONLwevXEPGTIkkYbeq88888zYfrOuXTTyRo0aAQDq1KnjLasJxiDp3Llz6EFbg9dhvBNfXeg8GFvjP//5TyItqduFoBBauS3uicZRRx0V23bFZjGxId7bTaSiewdBsDuAlwzLYrmI1DCOLxORmkEQvASgj4j8p2L/OABdAbQAsI2I/KNi/40AVonIHZZrdQDQAQDq16/fZMGCBRt0gyWUUIIfm4vubRuOxbM/uVNksIg0FZGmthFo0KBBsW1fr6kjhxGMEgYAXbp0wZAhQ5y9OT0823D++ecDiOjYFABtu+22aN68OVauXImOHTtaz+UIxzyAaGTfaqutACRHGlukMIKiIz3ipwHz/7//+7+8z/Xld8kll8S2faCVxOfJmClm3bJ+KTZjvrSabGX48ssvY/mS+sxtit3MdnTHHfExTJffjOTGY7Vr1waQFGvZ4qqwXet8WQbfAPnOO+9Y97OebOB1aKXZvM6nQpqJDQC7Iz7BORNA3YrfdQHMrPg9CEB7nQ5AewCDjP2xdK6/ynZ+A8+kn2bGbYnwedvW8PEGCoFeUZo0aVKl5r8lIw2vohCkiYSeC4VIDTRQmUunls7idgDdKn53A9Cv4vcJAF5BuSVxMID3JFo9mQegZsXfPAC1cl23SZMm4dozZ33pgIQwZ3h9pBwNci5cM/421/+ESyvAVQufV26XNuC0005LrI9XRkPIZDLSsWNH67GRI0eKSLITdbm1F3FHI58zZ45cc801snz58pxlMunfZJFyqbqQgEGaW2ID8+Xzfu2110Qk4qcEQSC77767PP3002E7cnWImgdjA1do9ArE7373O7nlllti+7iip5/3c889l/M6GiZhj1IJvWJSu3bt8Hc+zm/SdBRPAlgEYC2AzwFcBGAHAONQvgw6ji9+RSdxP4C5AKYDaGrkcyHKl1TnALggTeFMy4LahDTu6wlyMjTjLx/4Xhxi5syZBeefD9iYbN7EtUaBy3Jplj+1o1vtXdyEpkBr+IRSPh1FPv4oNXx0+EwmI3feeWe4zWVLU1eTCwsWLIht2yLGcVlY++80yViE5uXoOqPmqFBQo0PvXTVq1BCRZGdEVnKlWhab68/2GcJKZ+/vC/VHVZ2GKU5zMeI6deqUWDv3uXMXiT4TfC+oLYzd8uXLYwxSXRY6x9XxMERExo0bZz3HhA5IxPJpV3Y+qnAuDUgmkwn5Jza9i4g9GFM2m425e9MhGPQ5pu9OzVHxsUoJciZoaZgdFzt9zR61uW40y2+ChDZ2wBSHmZYYBzG6LLz33ntFJLIwaAmYHsiprOU9HnrooSJS/vn8v//9zxnmwYaiDDJkdhZaKcfOwmR06qhTegQzv+MnT54cfot26NAhwaxM47fTpxcxMWvWLPnkk09EROTkk08Or+m6Dtl6bIi+uRbC9elic6Wf9vPG/KxjI+XLqwPxmNR8F7R3b7McmprPzwRi2rRpiRGXRK409UOv17oj4H8b85XmuhbPka5dyGdiJpNxksfoEVx34iZIuuML73PhwCh7VEVrbL311nLMMccUX2dB6N7exqfX/itYobYXir+feeYZS3Um8xCJXnjC569B6wC0ctFGX2eZ2DB0kJgqVaokPhX051k+YDyVgQMHiojdMtIvGdP4wjAyQphmcto+o6gWZr62uDA0r12dw+jRo2PXs4HWSD6TxSwbP3fS+Legcpdpc32+iUSWo41eTtk6B0RbuEiNtMK9ouss+EKyIfCh28wv16fJGWecISJRJW633XbhhJyWKvtC+fElIG1ce5W2mfNUaVIs5huV9LVdoqrTTjvNOQ/gi8zNc9K8MDvuuKMsX77cWd599tlHfvrpJ/nss88SlgPNah0LxAaGVeAnF8HnbArZXJ2F+XL4aOIiSZGY7fNQp9FBq0ywfjiIaRlCWVlZYmJTlzVXmfPB1KlTE/l8/fXX1rRpO4tfhQ/OvffeOxGd24VsNhu6tasMLF68OIyu/ktGEATYkttKCYWjaHxwkg5N0glJLnfddVfOc1966SUAwNy5cwHYCVw1atSIbWv/kG3atMl5nUsvvTRWNhP0F0mQWOXrvMaPHw8g6WOSns6JbDYb3iOhqe76uImVK1cm8iPox5L5LFy4EEB0j9r/qA1XXXVVzjRDhw61luGtt94CAIwdOxZARHgzMWzYMGuet9xyS2KfJkCdfPLJsW0b2Ynn7LfffrH99G1pwlY+AKhZsyYA4Nhjj014Qqf3dl5HSwxM8lTnzp2t+bv8kNr2rVmzBoC9nabBr8KyKKGEEtwoGsuCKEQM89577wFwe76+8MILEx6PH3/88VgaHp8wYUIookorEjLpwPkIi0hBJu1bWyFr164Nf3N0ohdseiOvVasWAIRxRcwycWRkneqR5uijj06UKY2FxfwB4PPPPw/33XTTTQCAJUuWACifJzvvvPMSMTdsoEhv4sSJCQ/ievvKK68EEKdr9+zZE0BUZ6R/pwkJQU/jvngnGkuXlmsumzaNv3uMbbLjjjuGHrpZ/za6OgBsv/324TVefPHFWP762r1798ZPP/3kjZlCsB3ni5JlUUIJv3IUjWXBaF+jRo3ypjviiCPC37rnJrQVYYIjMfHmm29ar7NgwYLEiDxhwoTYtvl9yQhezz//vDU/LTkmli1bhtdeey22r2XLlta0PvgssjRyf0LPb/jA+BiMjvbMM88AiCT8JurXrw/APg8A+MtPYVTbtm2daa677jpvWc1IYYSeW8jnG58WmG4jVapUwQEHHIBsNhu6SnC5LdDYfffdw5gvpsVmgnkFQRC2KVpalYY0Syab689cOiUxxucYtjKQZtmKS1wMvMxz0lDDtTcpeAhF5GRsLE/R+YAcgI1dljR1qKF9cG5snH766TnTFBKMW8MmI6gMvZAGio1nQZCvYEMuJ69mQ9eVfumll8a26emaDl5N3H///SISvehUZDJPeo7OZDIhfZllWrNmjbWM69atS9ybLQBOWowZMya2beo/XOvtxIoVKySTyTgdy4pErvR22WWXRAfi0mrYXAiazNwhQ4aEedH1n61zIqmLx+iKLwgCKSsrS9WhNWvWLLGPvAo+G602pbjQx8Wht236KmVZpkyZIiLxOtDn6+2vvvrKWf40nRFjiRAmJ6dKlSphZ1R0nYX2oJ0PNKPT1zuTEq5japhwkXP4MLTnaBPsLL744gsRiRh5Nj+bl19+uYikaxj5jDgklXXt2lVEkkGAXF6/TVDJ6gusxM60ENIRSVqEGcuESlEd30Tnb6uTAQMGxLZJ9jPTsrPXhD92kLbya9+wfM5p7lV7o/fBVd9pnv+7774rIpFHeaLoOgu6d6dbc4pvbAF/SI8++uijRcRvNupKJgPP5xFcj0DmtUXiLx+DIOlwBL6XTO9L49mZ9/HYY4+JSPJlM1EIU9B1r77PKEIzO02Lhefzk0vTmG0vgy6/FsrZoGX6PHebbbYRkXgnwvby/vvvS4sWLcIy+qTwZIBSSOaqLx9YFlunoZWvhM9xNTs9viuushRNZ8GHoOnYpCrbRnldGRyJfA+MakbfS/ynP/1JRCKNQyF6jHy+ZV1y7yFDhjjP0aOhS0QkEgnh9Atp+1zQtHKtDqVoyQfqIyidNj+NCglExHgqhK3jcknReR2fx3GXhJ8vtQnWHV0DMH4IQZGYCap0WW79eWWDDq/pg34mbL9F21kQ2imKr0Jdno2owOR3pQ0UIxE6QItI5FvDFkUsFywPKpHmb3/7m/VcWg0+9+/6Om+++WbqsvnqhWBj1Wa3jpiVpmw+3HXXXSLi91WxIZN9ugMwoT9b01hPhCtYUhr4BhKfj5Fc0OW3bBdPZ6ElwiLxiuVIHwSBrFmzJnxg/ATQIxBRvXr10FxnBfJaNG21DL1p06aJh8pYm2mgR2TC9JVBJW0+DU1PBttGMg1OdNosCRF7WD2mNV8KxkjV4GoRYfpnyNVh8N45SWd7kZiGcxd6glkk2aFzVYfW5qhRo0SkfOJ81113lQULFoT5MNgT24Y2503wGMur70/HahGJLBTt+Afl/mmt19HzP7bA3S7wHG19pu0sSqSsEkrYCKhevXpe3JTNiaIhZRE6FsWnn34KoJwE4yLNaFEVPUZPnToVQJxsQ0GTjivx888/I5PJYOLEifjss89iNGHGbGD+jzzyCADg5ptvxiGHHGIti8by5cvD+3viiSdiHrZd9+Wir5u47LLLEvvyIWEBwNtvv53Yx/o544wzAABPPfVU7LhZP6RYa8o5EBHZCNYP6et8VrYyksDlik/ig+ueTQo06535NmvWLGe++nm7PGibQsVcpKyXX37ZeYxltB2nCM9VL4XGESlZFiWU8CtH0VkWjMtAkEqsRzYg6nUp5jnmmGMARLE1fKCEXI8UPgwcODC2beu5ORrmog5/9913qa9rglHQdNQ1LWs/66yzwt8u+bovypuG735YL3xWPmiJdxrhHWXmNho5QVm2Bn2M2K5DiQGtJNLJb7zxRgDAnXfeWVDEMcAeRU7XIffvvffeOfPTLhBs+VDIR8q4vq+0+MVYFhdeeGEY3KWEzYdCnAO1aNHCqawsdtSrVw9ffPHF5i6GF0VjWbB3bNiwIYBohKMwy5SBa9Aa0TJxjrZTp04Ne1mKkjRsMTD32GMPAJFjE0rhL7/88kTabbbZxps/wTkAE3379vWeA7hHfDrksYEj2RVXXAEg6XBG1xsQOYCZNGlS4pjGzJkzAQAdOnQAAJx00kkAyr2G6XPTzjtks9lwjujEE0/0pp0zZw6GDx/u9VCWRqLO50uk+danRJyRz7p37461a9fGrrf77rsDiO59wIABAKJoe3w+dPxjwmXJpXEGxesVbCCkWTLZXH82bQjX3wnGgbCBfht/97vfiUjli6Dont21HGpCL9/aHA0TZOMVUl6ewyU1eDgC+TA5dXAnIh/hF5cKfdfT/A0fdMAjre/xwVcvRCH1Tz+vaZa9qWmht28dc0SkXHtjA3kXuoy5wlXYyopi4VmwQZCJt/XWW4tIREziQ7E1Wrp/97H0SERiPppabTZaM9qTSKQBSdMw+FDZsYwfP96ZVnvMdvEY0lxv5513lvXr14cxVMx1ed0Zk2PSo0cPEYkYhiJJngDr38YozCWQsgn+WrduLSJRgB3N6JwyZUrIp+B51PH07ds3UQYNMoH1y0XN0fvvvx/b/9e//jVkPOpy20hxLJPNW7vtuiZsnYRIxAWx5UNdD0FekI+PQpCtTCp70XQWBF8uinlYOSNGjJBHHnlEstlsgn3J8Hw2haGrQsn+TNMB6NFpzJgxMmvWrESME5EooAxpwPQM7oP2Gu6DfrmYv025yrqjZ3Oeo0cl0+O1i/psg+7sTJIU86KGwlV+/Z80apHohdCqTD6zNBZBmjRaH+QLS8lgT3RBwI6NbZiKYl8kNN052aBp9777+P3vfy8i7nqp6CiKr7MgBZoiJLPRtmrVSkQisZkOG5APTJahiMixxx4b/ma+LAv9KPh6cyo827ZtKyL5NWQGa37++edzpqXKUYdBSPOZcN111+VMwyA8vvL7NCsicUo486H6Vu8n1q1bl8inECFcPml1neXzzOrUqSMi9nCMWiWrzzUj6JkhF800+fhz8ZXbbLNF01kw/gVvnEpSF7p06ZL47k3zsPPRnjAAjp4/oY7E5y+C1ogr/wkTJoTH+MmVxj+Hvlfz/5NPPum9H5t4Lu3IT6q9zRJzxS7xqWjvvvtuEUlHoWfwH6qA+UmpnRNNnTo1EQUsTTQ5xjvJ1X4ymYysWrVKRCItC10O6LkpM1Qj8dRTT1nzzWazsm7dOslkMs4yuCzgIAicMgeqovnsiqazyAe+ScONgXx4+WnARl5I9OxCrpMPcgnFGHtTxB3r1DepyOBCLJttHsKlzTDjn7qQb1sS8Ze3MpDPc3B1CjbRYb7Pt+g6C04qsYelecptMyQev9MLeSn0/Ib5vedSFKb5VtbHevbsKSLxUcW1ssO5BB01y1amV155RUSSYiHOYZjn6DioLrWu7z44iWyC9WFz6KPzyjVq+8L0cbLYZpo3bNjQeg4/7bSFaisLt2l10lOWDXQnoDs039wUPVVxPibNPBk7ZVpRnMOzgX5dqFYm6LOF95y2s/jFkLJKKKGEjYNKI2UFQfBwEASLgyD4yNjXIwiCL4Ig+KDi73jj2PVBEMwJgmBmEAStjP3HVeybEwRBt0JuKi00CYv/L7zwQgBJ781AktTi8wSu0aBBAwDAbbfdBqA8hkM2m03EIAEiOnYaUpD2TP3KK68407pEYj6qNclBtjwymYzV4/Y111wDICKgkdK9xx57OO+JZKYHHnggcey+++5D9erVE+WvWrVqbNukSbvSmNHlctHTTz31VABJOjwQefzm/TCOC1WktryZ7/bbbx/bP2vWLADA008/HUsHJNuY9jBvEvV0/bjapUlSbNSoUew+CDOuSj7IaVkEQXAkgO8BDBOR/Sr29QDwvYjcodI2BPAkgIMA7ALgNQC/rzg8C8AxAD4HMAlAexH52HftkmVRQgkbH5VmWYjImwCWprzuyQCGi8hqEZkHYA7KO46DAMwRkU9FZA2A4RVp84aW5vJ/nz59cPXVVwMAxowZ4z0XAH77298CiEZv9sJpQPk0Rxydf1lZGebMmRM7pkdpH126a9eusfw4kh133HEAotG8rKwsTKPjfnI/KdJAeQzVXr16Oa+rY2hoSwOILKM0QiodR5bUeeY1fvz4cMTVcVq0ewEb0tCvO3bsmDONzouRyO677z4AyXtlXNSuXbuG9dynTx8AdktF50EZfj5lygVSxk1oi4JivUJFcKkmGgHsDuAjY7sHgPkAPgTwMICaFfvvA3C2ke6fAE6v+HvI2H8OgPsc1+oA4H0A7+fjb7CysDHiMvjAZb9ixqauU3q4qmzQ6xb9wfpQu3btvPN/44038j6nMoCUE5yFCskGAtgTQCMAiwDcWbHf9tEqnv3JnSKDRaSpiDStU6eOswC0HvidR/mtDZ988gkAf1RvWhgcXf/xj38AKO+FtSMZV89MgRkAvP7667HvR4rCGO+TowZjlAJRHMtcYNxLE4z6pcGRr1u39NNE2WwWixYtiu3TEb95r7ouJk6cmBgRbVG/bNcEko59stms07UAn4uOefrEE08AKL935ktrrF+/fon8db3re6Ijm/PPPx9AFGXOBO+Z8Wm/+eYbAMDIkSPDsriiiTH/5s2bx/a/8MIL4e8DDzwwVjbGbCVMqTrj3bZr1y62vcFI06NAWRauYwCuB3C9cWw0gEMq/kYb+2PpXH9mvAoSfLicqinEFMXYoL1X+/gYpGqTCela2lu0aJFMmjRJRKJlLF++c+fOFZGI2Xfttdday2gDmaK+IEpcgkUlcAM0zdl0Q089xmGHHSYicc/p2h09lwQpuNNLzGksjieeeMJ5jC7788nPdS5h5kHSUj5iOdaHi2mpY7SIROxeYtKkSbJ+/fpUDEyW1wwxwfNq1KhhPZfLxwQqk2ehOwsAdY3f16B8ngIA9gUwDcDWAPYA8CmALICyit97ANiqIs2+ua7bpEkTZ4UxYpiIfa3frEhi0aJFImJXiepQhNRyEGY5tKBMl9H8tNDH+JBtXrnI2x0AACAASURBVMMJcgv44rtEUCJRkCKC/AreO/kn2Ww2wasgzZt8CF6PwWhMaOam1tDccccdiXOYv34OzMPs2Gjif/LJJyIiMnz48ER+hGbIMv9rrrlGRNwcC1sZSMc3wc8Mlo+UdJ5DjowJPld2Erq+2OG+/fbbiWNURe+6667O8jMtGaI2bo/W7ejrDBs2LHGs4j4rp7NA+erGIgBrUb6ScRGAxwBMR/mcxSjVedwAYC6AmQBaG/uPR/mKyFwAN6QpnEnKco0aplrQ1HGY5+j4G3feeWd4jGHmOFJqZDIZyWaz8u2334YV3KZNGxFxB7fJZrOJh6kFQMSf//xn6/mFwjWaiIg0btxYRCK6NV8uRj6jp3MbJZkdpO6wfKQpTVoj+BICcDbwfIR8uo5t4L3pNAxFyXJsvfXWic4sDaOW9HE9n2F78UmKYkhDWhu+uZD58+eLSGRhs37YcdksSnYODF3hQqVaFpvrz0fRpS+AHJUQ27Y1Js2w46jFh0JpthZoufJzQfP/ea7t04WMOzYIm06CnQ4bmvbt4BrNbfBNxlFN6cqHyl4bGErA7BxcYNCofJBP/TMsRD7nsk71J6MP1JNwEDPvmZ9jukzaqs3Hl4evU2V+2vrUKLrOgj20qTugySpS3tjoE+Htt98WkfxCyGmOPR+2OWq5PinSzBO8/PLLIpLsnHSeK1asCOOsUrJciKWhz/nhhx8S+xiOj2pR3g+/1U3oIDc0q7U1JxL5w9BxR6j/sEGLwGzQwjnOJZjmtUgyGph5ri8EYS74/KLwxacymUgTelLDtwqo561sQaRoMRLdunXzXq/oOgti6NCh3hsvYcsDTejKgnb8siWAatlC4HOE5EJlLkcXXWdBa0HDF+2ck6BcifDBZSXkY+pyRLadQ8EPP3OYhhbTqFGj5JZbbhGRyCcCRwjT8YsLFNalcZhjC6Jr4txzz03so4tCguU3ZfrZbDZmgel60CMmAHn99ddjXrjSmOIuwZfvWdGqSZPWla+v3ughi5bLtGnTRCQSAdqe4THHHBPLX89zVatWLfwc5qS57iTYtunoxlZ+1zaRtrMoCclKKOFXjqLw7v3JJ5+EpKI//OEPACJSCoks8+fPD9Ob0bwAeL07a9IRvYWTCEb6N2m0ZWVl4Tkk3qQBy/voo48CAI466qjYcZafpJ4RI0aEx+g1XKfNZrPo1asXBg4ciGeffRYA0KVLF+t1ibKyspAubu4zr23ip59+CunkrVq1StDrXZg/f374HJgvPYO76MtfffVVmG+9evW8+Zvo1KlT7D6YB6OLmfR+es7WZZg8ebK3bCYYcc6GBQsWAIgIaMyPgjVbvXEf01x//fXW4z58/HFcXmW7jyOOOCK2zTrOl/ZdsiwU6tSpE7LvSijh14CisCxsYNxMTWHdc889w2hejNGhBTu2npRRudgj+2jjubBw4cLEPpaF8SRYhldffRVAJPbJRzRUVlaGKVOmxPYxH1d0q2w2G8Z4JWjl/PTTTwAiAZU+FyiP3woA5557buw6OoaK63wgGd/ULCupzvfcc08szSWXXALALm/X98o6mDp1KqZNmxY7xkhsrtHUtt8lCqPlsvfee2PGjBnefBiT1zeK0/IiaJ3Y6pR0eG1NlZWVoU2bNrG0Z555JoAoZm379u1jx3Ubyok0Exub6y+NK7QOHTrkTLOpUYiHrsqCngDTS3mFQk+O2RzSFgJ6lspnGZq+SfUSoQ8DBgxInXbMmDEikm4S1DfBLpJOdGZzjVcZ0J69Zs+ebU2HYlsN0S7t85nVpsaEKwXa5ZwPzH/06NHhi+fSjVx55ZUi4qcqc6WD4Mt9yimnhPtcLyJfKLIObfnYvGG7wPxYptNOO01E7HVKD+qk1uv1fpt6lvkwUI6N5k3wnl3MV4Le3c1jpobIzKNRo0YhbV93oqRNs/yNGjUKuTVMS3o8V2tsy5W5yG/cHjlypLRv3z52jOXWeSxfvly23XbbcHv//fcP03CVJQ3pTnfAtvqvWbNm8XUWuvGYlaPjR7hgVjA9Q7Pi6tatG0ubyWRCz8psoPqB2Mg/GqT2Dhw4UEQi8pEP+h7TjLaFiLQ0CcvnS5RhFvJxikyaMfN1efs20/C5MLCSDv4jElkU+++/v4hEL10mk5ElS5aEfkhN8EWlMI0EPtYB/Wz6mLq6Tm117Gqn77zzTiKtWW4TPktDPyMS93yDpu4sdFmLprNwhR70KTDhoDzbQEGUps/ql+Ktt94KdRy+l5gjsIbrRXR5wjbPoSXhe5l1yATC9HxNToSuF9OU1kxNre/QfAsRkUGDBlnPcT0zU/WYlgtwwQUXOK0NxjQhTN4GgFCD4cvfVoZc1uu8efMS9cM2QSstzXWIZs2ayYMPPmiNRKbbmu8+tF8MptVWrUh5uy+azoKiJpdUOQiCUDZNuCo2CIKEqIYKS3YWFFvZYjlUrVpV7r777kRnwf++YEP8Tx2A/lTxkZl47lVXXRUer1q1akywpj+tqJSsWrVq4n51w3Y1PDO2hosYduSRRyb26c5In2uS10jZ13FPGGZBC7G6du3q1Nm4ts19/E8Nx4cffuhMqz8HGW3MB19bYN465mvaDlMkGhSo0mVaPqu6deuG1hGJgN98840138MPP1zeeeed4uksiEImDXNNPtnQsWPHxD6ahZRy23wS5AtGEPM1jHzYo/lSoNevXx92MHxx0iDNJ5GOp7I5J3wZj3ZD4HsO9DWSz7PaEGg/LvnAVRdF11kQlBq7GuDrr78eNmhOummqL9G3b99wXy43aGPHjg1/U8DEMnBGX5vjJrQPDV/MUJYpzZwIwc5Hx5HIhzbtAy0h3VloK6JRo0YJyrarDObniJ4rSKOmdHVcVO3SirOd64Me8TnocJuuCRj3w5c/o8+zDdrmU1xo3LixnHDCCanKPWfOnEQdMuYsxX60RvTzSNtZlEhZmwnVqlXDDz/8sFnL8P3332O77bbb6NfZc889MXfu3I1+nRIKQ1GQsiZPnpzw8kz/hzYP3vRDSJp3jx49YudqmD4ySXbp2bMngMiz80477RSmX79+PYCIaOPKd/HixYl9pHHznEI6infffRdAebwPxqf4+eefrWXRMUFMvPHGGwCie95rr72s6V566aWcJCbzurZ4LACsHQWfn+m31JY/0aZNm5AMNW7cuMS1TdBzuwmdlvT7FStWAIhIWyZIWiLZi9vMi35NM5lMWF5TfgAATz31lPPeKCVgPBvTG/c555yDffbZB/Xr18e0adMSMUaYlvtZNpIMgYjsxTT//Oc/AUQEwTp16iTinPhQsiyKDB06dMDgwYM3dzE2KbLZLH7++We0bNkS48eP39zF+cWhKCwLIOpBaS5TQEaKsg26F/ZFFXNRwvnfHC1pURD0vsyypIkyZmLZsmU5rZ9cx3Qa3VFcfPHFibQuURhHV1vULF1PWoRGoR8Qt8aAyHoglTubzeKrr77y3o+23mjV2UDril66TVoznwljmPjo8BoUF+q2xvtbs2ZNuO+jj8oD9v39738HEHn1Nr23u8BIczzX1l61Z3pNRfe1ER3Jzmyn06dPz1m+8LySZVGcKCsrS+WCHwB22WWXhGv5ysT69evz0r4AwOzZs7H33ntvpBIl8fHHH6Nhw4ab7HpbEorGsiDY21KSraOBAdHocOmll1rzMHtUpjVHBwBhVDNGzzJBNarrO37rrbeOlRVwW0Cm9dOsWbPYMdfoRxx33HGJF4kvIy0VdhQ67wMOOAAHHHBAbJ/uKLTgC4jikujYF5RX2+KwMp4Hy9a9e/fw2BtvvIEFCxaE39UU4TEyma2j0LFR+DxYx5wvoNgQiKK4caRn5DCNTdVRjBgxArVr1wYAHHzwwbFjt99+eyI970lbtcRvfvObvMugY7OkRpolk8315xOScWmNS1M+PPTQQ7FtEli0S3kT2ouyDR988EFs29QoaCYkob0jFQLtrdwGuh/0efvW0P4pGUPFBmoubOQ1F0hG2lggu9RGtNJ47bXXRMTfBjTo65Nu8Bg3xgbTP2xasE3wPmye37EBcWFIabfkWVw8C03HZpAYG3r37u2pMjtM93AiEYW4T58+4T7SaPlQzz//fBERqVWrloik033kQ1DS4jDt7NdMozkN+VwnTVp6uNYBiMzOgkzBfDooOrQl09T2MqxatUpEJBRXUWnMMIVsC8zL1hEz3AHvlWW1IVdHTh2R6a7RdY5tP3kgLK9Nb0Oavh6sqBy1DWJ8NlpYR1AfYz7voqF7UyRjg6nKc4FEJUIzL2+77bbwN60NCr4+++wzEYnIVGl0Juw8bKIhgpJl5p8GbHB65KeaMw0YbMgH10tuCpvYcbHh0YV/mg4njTUFRZe2Cbs0KCm/8847nWkY94T3yEFAx4uxlZEvoUmd95HqTGj3/zaQSMe24eooTNB60gQ42zmUSjBN//79Q02USBFZFmROXnrppbEKYKWYL7yGKft2AUofQYvCZ+7ZXOVvKJYsWRL+zsdPg+sFtDVmXxS0Qq+T65oi/rrMB4sXL847f1oB+dQp4RqhfaDj3jT56kh6vjKm/Ww1XQVQlauvq2O0FE1nQXBURQ5FqelWXYuT2PHw3LKystQiHrMR8Hw9snM/P0s6dOhgFYGZ/02FYP/+/WPiMEKP+J999pkEQRBTlLpge5E43+ASMpEezHLUrFkzTMMwhVpZevPNN4dUe1LyNWhNBUGQ0xLRArxMJhOGQ3TBFieGQjRdpxdddFFsm2EHSct+6623Ev4m9CieyWQSAwep4UzD+SVbu+L8VVoMHTo0zJcDoS3fbDYrQRCEAx/P4f1oL+VF11lUJtK4y88V8s0HmxLzl4B8REpazZrmc6FYkOYTmKgsT2W5sCFivbSdxS9m6ZTQy4rNmzfHddddF9tHiq1rbX/o0KHhbxttGUDCt2I+ZTKXHknRZv6k9hYC+qMkTLq6XtKcNWuWMx/X0u+kSZMA2Mll+h779esX2yYNH4gIVC6SmrncR5qyJn0RJFoBkQftF198EQBwyCGHxNLaaPj0ZK2Xg21yAYIev0nb18utpOrbyFOaNt67d2/ndfL1rg0gIX8guEwNRD5C6ef1wAMPtJ6TL0qkrBJK+JWjaEhZWojzxz/+EYDbIjDBkYdp9EhkGxmYNk3+PNa2bdvYftPL92WXXQYAIb3ZpEWbeWiCjlkG/r/pppsAxGOP8BgFQXo0JxnJBD1la1o8t0nsuuuuu9CqVavYsTT0ZVo5JKnpOrWVj2l0WXyjr8v7tnkOSXy33XZbLA3rif9NIhdHaQrSXOU3qfW63nX5bbR7fc8Et9nWAaBdu3axc3zSAld5uT1z5kznuT78Yi2LRYsWhSrTEkr4tWLKlClo3LjxBuVRaZZFEAS7BUHwehAEM4Ig+F8QBFdV7K8VBMHYIAhmV/yvWbE/CIJgQBAEc4Ig+DAIgsZGXudVpJ8dBIFdz+yAFhJxhKOIyMSee+7pupfEvkK+G/VIcP7558f2U+xm5s+OTVsqpqSYoJWgJd8cSW1l5giq869VqxaA6LsViEY5XR8PP/xwIl+C+XIO4c4770ykofRd08k5H0GLwyYHZznpIsAHrXlxPcPjjz8+jHOiQco1509IWweAhx56CECSBk/Y2hGtjL/85S+x/T7LlOXm/dx111050xItWrQAgLCjGDVqVOIc17V32GEH53W8yDUDCqAugMYVv6sDmAWgIYB+ALpV7O8GoG/F7+MBvAIgAHAwgHcr9tcC8GnF/5oVv2v6rp1mNUSHCLCBS3YMrrypXKD5oJmQlV2mNCSyTQ3e4zPPPJP6HN6HLfAvoWn32lO7DVxiL6TeuYQ7btw4Z5ohQ4aISLpVivfee8+6n8u5PtDbWN++fXOmJXTbwMZaOgXwAoBjAMwEUFeiDmVmxe9BANob6WdWHG8PYJCxP5bO9mfrLNLExdCUZzaInXfeOZGWx+6++24RiVze67xMLFq0SETiRCoRSTgONsGlRgbI4Ro4Y1OY0D4su3fv7syXRKgzzzwzVl5fI0UF94Jl0a4FV69eLatXr445EbaVUyTSe9x1111hWtLtuZ5/0003iUg8ngfRs2fP2DGCee20006Ja9KNv+5wfWC4CD6HNB0Ky0BattYY+cAXnZRwpCCmMZbJihUrJAiCmJdvPs8RI0akLreGJpl98MEH8vbbb2+czgLA7gAWAtgewHJ1bFnF/5cAHG7sHwegKYDOALob+28E0NlyjQ4A3gfwvq2hUDjGCkkTaYpptX/KQmDqAXT++WDBggU5z50/f76IJDu9NOA5fMmpK7GxBDVhzNfR8KVevny5iER+KAcPHpyzTOecc46IRJ0S85o4cWKYxsX18Pk3Jch/8NUT9Tvk0biIaTZoMpMNDF1R2dqcysiDQjLGsiEqvbMAsB2AyQBOrdh2dRb/tnQWTQBcZ+ksrvVds0mTJolR1YxIJeIPXMPOIY0ugmDvTnq5rRHVr18/Zz4jR46MjcwUQxEmN1/DFt8hF9I0Fr5waV5sIk0HkqsstmhlBDsMWglprsNzPv30UxGxP1++2C6Ke5rrzJkzR0SSzpZnzZqVSEv1KuuL16XF6itDProaLRy0Wc2ujs90kGyiUjsLAFUAjAbwd2PfZvkMcSGTycRMyzp16oQvvg9070+kCQ6TD/SD0w0jjULWZW4fddRRTmuJdOAgCJwepSlccmloWHYz7oemuPNz6sUXX5TVq1fLsmXLwnv87rvvYmn1vZtxSUTKTXWtAeInAJHNZhNWBk18LcG++OKLU8/dpAnJoIMZ6RHaTKvp6vz8ufbaa8POjcJD1gvnQjRd/sorr0x0WPl0MPzvsowqrbOomKgcBuAetf92NcHZr+L3CWqC872K/bUAzKuY3KxZ8buW79q2ziJNJel5DV9DcE2Q+hSvmxqFfH4UAt+5dCNP3QthG2UrE2msQnZgZiQyF9LUT5r5hcrE9OnTRST6HCy0LIceemhB51ZmZ3E4AAHwIYAPKv6OB7BDxSfG7Ir/tSTqXO4HMBfAdABNjbwuBDCn4u+CXNe2xTp1KRvN8Hb8Ltc9qy3eBHtsHuMozgdo4vXXX5cHHnggkZ8evdJYRD4BGwMsd+vWTUSiIECMg2HrPEaNGiUiSWm3L+QhoWNr2hoZQyjqfOhExgTjk2633XYiElkw/LxKM4rzv28VhBPKjOPCe7X5FWF+9CWhJd2mYx6KyQiuptHZjY5OLpKMLcI2mM+qFM9lXBITurz8rzU5XIUxkWv1KW1n8YslZZVQQgmVg6Khe2sCjCaa2Cjbet+8efPC3yR3aZILr0NP1CR2UZSTBszTpOm6sNVWW8W2lyxZkri3fL2Fm9Det3v37p1gvGpy0yeffALATnLSNHUXldgHeve2gd6rdT4uSjcQEc14r/RhOWfOnDANfalqotguu+zizJfxQIg0Aqynn346tu17dprSTnCbBC+bf02fp3oXSJZ77rnn8j7XRMmyKKGEXzmKxrIgdG+8cuXKRBoeY7wHbpOCS2/Qd999d+IcV14mXFbN6aefnrr8TZuWP5OPP/4YQJxmTHCU1aNIlSpVYtvffvttKI5jWorAOLLpGB5AXIgGAJ06dQIQxa0w8dprrwFI3juvS7GYCcZ40ZG3GjRoAAD44IMPwrT6eTK+B60gn8XiEkyxLmbOnBkeY+QxbuuIbaYlwHbC/C+44IJY2qpVqybKsnbtWgD2+nCBZbFJFjR03BObVZLLK3wh0oYY0kxsbK6/jeX8xgf6aiTOPffcTV6GtPD5+tQgwzMf/FId2tjIfMWKjh075n2OnmBGsTi/0U5R6EymWrVqAOKyc35z61EwzehEHHrooQCieBumKEyXweWwxQYt9DFl7ATzW716tTUPfR/NmjULrYThw4fHjh1xxBGxbV/MTULLn+nQJpvN4sQTT7SWyQY6I/LFWyVc3+B0QLPNNts4z3XNJZjRzniProhmnDvyzQXo+Y3DDjvMmTaNyM1VblPs9/nnn+Mvf/lLzAoGIrEZ51VOPvlkAMDZZ5+N/v37o0mTJgDK78d1T7RkunbtGrogSIU0Pcrm+vNZFi42mglNmiEV1wa93Eavz1z+M8ElNL2MxSVPG+j3kstiZ599tjMt7w3K03Uh8C3d6RHmkksuyZkfrQ3qYnz5c6nUpcMwORS6LLxOGo5JZYvwNKWfy6CaIGYDn5UWt6UB2cqdO3cWEfsSKtSyNu/dtlzMZ0ORmfYIztg2KBYfnAzWQtoseRZkufHhmI56WaGacUc8/PDD1v0iSRKQSe0+44wzYsc0F4P52oLp3Hrrrdbr0QGuSBSYhg+ZQWa4fcIJJ4RpdaNh7AnNq9Av0tKlS63MTFueAELOwVlnneXN94orrnAe4zNi/qZYz8WF0f9/+OGHROfP+uez53XIip0/f36CC0Oeha4nU6ClzyHT1MWrMaHp8QwFcNVVVyUcLDMfPShotqaZr+t/ixYtnB2JLpMZI2fx4sXF01lokKXn8iAtElU+RzaqFOm12jYSMXiLC9tss00iGMy//vUv7zkmSPZiQ9Av6m9/+9vEObpx2khlbdq0iZ3DtKQS8xx6Ed9tt92clGTd2GxlefDBB0Ukqi/uJ7PTVre6s+C522+/fYIQpjU5+v6IBg0aOF8GWxn0vbr+m3oeqn+5TZGbj+jWo0ePWFnYwaWxfrS69ZVXXpHRo0fHykTo+mfd2pxRM55Kq1atYuUniqKz2H///Z2TbGT2sSMwkQ9rjpNhOoAPQdZkPqCq0ixLGialBtPee++9sTxMpDGNc+Wfpkxs/K4QeL5zCBuTkCYy5dOMFJar8xbxd26Ebh+anWmCAxHzTZM/kaYOKUOojM+mQvyVaI/sRFF0FgCcL9sDDzwQ2xZJ+pfwmW76gekHSUugS5cu0qtXr/CcfffdN3wJeD1+quRjpk6ePFlE/PMRFIk1a9ZMRET+/e9/O9MS/EzT+cISz4NloRMX0wLQaan25Tl8uTOZjIwfPz42qrtGetYLR1/bc+A2BX18YbPZrPMl06O4uV8/K5d6k9sNGzYMFaP688B0aKPrl+2Fn1g6/2nTpjnLvf3221vrwCyrblOuNlYRjlBEImuVOifXgJW2syhKUlb9+vWtqw2/NJxyyikhZ2RzY+XKldbI9SVUHp588km0b99+k1+36EhZJPRwKWrNmjUAypf5giBAJpMJl6S078TWrVsDiFOg9fLVwIEDY9uu5THbkhTz4v6JEyeG+1neb7/9Niyvmda2vMXlNy63Mv+PPvoolm7EiBFOIg5ja9jy157LbWlMktLll1+OmjVrWvPg0mZZWVnogVovF+oYIySkmc+BnXudOnViaffYY49wGVVDE5Vc+MMf/pDIg9fVy7vTpk1LxBj55z//CSAijpnEKD4r7c1bt6+vv/46/M002sP42WefDaDcP2i9evVCn6VZIz7Mhx9+aL0eUE4Tz2Qy4TPRz0H7e+U7lBZFaVmUUEIJ6VEUlsXkyZMTPXUayqorDck1q1evdo4I/H/SSScBiARIZu/uGkVIjBo+fHhCSPToo48CiKjcPMfs7UkAo7dwFxjP4vjjjw+tGFKsmS//t2nTJlbWHj16YPz48dby0/qxgdHAeI6mPJt17hpduZ8R4fbZZx8rTRnIT0Snn7emxQOR5UKimY4UZ2szHHn5aaDr1mwTBOPEaOvBTMe2p++bHs4Z18UGWii6jrt3755ISwq6zQqpX7++8xoulCyLEjYZWrZsGXZUJWw5KArLwgR7RdK8OUIMGTIkTMPvaltPam7z2/Pqq68OxVN6LkHn8eabbybK5PpW9sUZ1aDwq127djG6r3ltls2UmD/33HP47rvvwm3Wi4ZtxGS5W7ZsaU27fPnycB/p5IMGDQKQpE0z/mrfvn0T+dACIthR2MRzui7N2KkajB/KUVzj8ccfT+zjnBe/+YkRI0bEymxi7733BhA9I4IUcfMcWl7EokWLAMRHd+3CgOe4ZAOXXHIJunbt6iwfgLANLFy4MIzH4rMQNTh/lApplkw2159JyrJ5ANoUqGwq8YbA5+RXg+U+6KCDCr6ezSsZlzQ1e5XeytNAL/uZDn21mz6bV6qNgTTPGQW421u8eLGIpOOn6CVT7Zx6YwHFwLOwMTi5Xs0KXbt2baKS2XD5oPIBXxBb4+E+10trUroZK0N7H/cpRXfdddfY9o033igiIqNHj06kJdlo3rx5ImKPu/Hll18mfGaKRKxIzZ3IZDKJwDYuroo+LhK59nORj6gRKSsrk1dffTVRLhGR4cOHi4jIkUceKZlMJhYagGEg6EaPhDrNxaESc8aMGQl3jHRaa5Ytk8lIgwYNwrLddtttIhLxIKgTWrNmTbiPOguCdPHnn39eRJKcH5Eodgn5JeSFuDyQm2DbYLnJJtYMTvO6J598cuwYYbKHK3gZxdNZ2AROrpHA5nGZyIf1xrSUqJvXowt6ggSxNCxNTcApJJYJiT+kcNuQ5l4vuuiiWFpdbh2UxkzTuHHj2Lbtnl3+M235EhTY2URULujOQlPdbaDr/BdeeEFE4paSr3wiEdOzEKbwhkLrjnbccUcRSVqBDCAlkuwsKIyjTqZoOotcL6C5nw+ZlaPVm2mgxWdkWnJUM8FRhg3Bpk0ohOatG6H50l177bXWuB/Mn2xSjh5mI9UvL1maNqq7bnzshE866SQRiViYIpGzWuLEE0+MlclVVpF0nQ4xYcIEEYmeia4nGxuW+b388ssiklT72j610mhNNBgoCQ6quKleZrlpxTHtUUcdFabp1auXiESBkwgqRYnnnnsuUUZdD7R6XIrYouksCK3803LwGjVqiIYexZnfmDFjEmkJ3VCOPvpoZ1rqVuj9OR8rgQ2E1+MIkc1mnfRf3QgmTZoUms86re9zQYPHGL3M9gLRY7crX0Y8I5599tlwn9aEaPGYr/x8sZ5++ulEOqbhZwPrn+dMmTLFgX6K9gAAFoBJREFUWV7G6HBhzJgxoQpYm/o6LxOsO53WjGGj49m4BjN2JjYLRovzbrjhhth+XzkZjc24fnF0FoVMKm0qpAmpR/CB22JDbG7k4y9DNzyGWLTBNWc0cuTIxD798q5Zsya27fvkItLcB8MtUt2a5lOiMia5mUeaWL355JcGDMVAFKXq1LQseIN82D5o3xHsyTOZTBiLg9Cjti8cItO4gtn4VmxcI76tsbITMn102LDVVlvlNN+1rN4GSvd1x7x69erwN+/d9tK6Ok0du4SwCchywVZPNuGYiRdeeEHatWsXXtMEzftsNisrVqyIHaO15rIobNCWUPPmza3HRSInN67OjZYqsWjRIpk6daqIRBP85jxN2rkTXUaUxwP6dQvJStiy8fHHH6Nhw4abuxglVKAoSFkrV65MkHVIlNGEIhOvvvoqgCTd+Iknnohtt2rVKiQduejF2WwW06ZNC38DSb+N+lyTQKMFYy5yDSnFZl5Mq+vA5l1apyHBS1+vS5cuoX9RkqNc12nZsmW4j+I82z27yGlLly61lmHVqlXhc8x1H4R5Pf7Wz9cFEqMAhOIsLVgDgLFjx8a2GW/j5ptvju1nLBPTk3f//v0BRFRzk9gGIFTssuznnHNOeEyT74ByD+Osi8cffzzRBklGbNu2bWy/DzNmzIhtpznHRMmyKKGEXzmKwrIwceONN8a2d9hhh0Qa3VMyshPhk2Tz3OnTpwOIZMnE3LlzAZRbFRw9LrzwQmtZbSOeaxT0WUjEKaecEtt++eWXnWl5H2YEsbKyspiXct7/XnvtZc3DLCt/69E1FzKZDMaMGZMzHSXiLtGZDatXr8Zll12Gnj17AgBeeOGF2HEdUcwE5fLag/rOO++cEPC5LC7ur127dsI623///WP3Y4tvQzBfukdwCfBs16akYPbs2c60uSwvTcfPhZJl8QvCjz/+6NVMFIovv/zSG85vS0eLFi3CT4a0OPXUUzc4nJ8NixYtQt26dZHJZJzhBzY3ysrKYiELisaymDRpEoCoR/V1HhT8UKDDh0VRjw8HH3wwAPt8gEY+QjGWhfFNdG9PhzY2ibce0fjdSzlyJpPB66+/jpUrV6aKPkVrqVevXrH8aE25BHiAO06mLUapPl/P6Zx33nkA4lLyevX+v72rjbWqOtPPe09bo+Wj0GrlIgwwNEZqYkUzYRhE+aFYfvgxA1pjldhJUFKlZWyid0iIqDUqOiADGmcEA3UUvyrgj4bea4i3elMcai5X0aBXCk4vX2rV6Q8zjpd3fpz97L32e9baZ5/rOfecue4nOTlnr73W2u9Ze+13r4/nfd+JqTy8d74oaQTbkorCNQbk2oStn2Xs+pXrKsC+4W1dFocOHYrL0Nhv2bJlABKjMV+bWpP30BqYDxw11zo6AIDjx4/XXAZA9a1TAJMA7ALwNoB9AH4Wpd8BYABAb/RZ4JTpANAPYD+A+U76pVFaP4Dbq13bZxtiWXX0Np2FhQsXpo5tjAV3O9VuZ9E+g/vyWfgyfAUXNr6JtQvwgTYP5CuQEBXa5vXJUst2JtuDNjA+XHvttanjLKMwbnf7fIdWw+OPP66qiTPeWmjYtYD0aDIiXdksDZttmcUPsd7hbfv7OBm7d+9W1cSgjyBD1JL0VJP2oFmCdWKMevEsAEwAMDP6PRrAOwBmRMriF578MwDsBXASgKkA3gNQij7vAZgG4BtRnhlZ13bp3tbK0Yeenp7UcR66ru1YpIyvWrUqVYdqYk8QMiTL6tjk41uZ8lgWct+dlHNXJvuguzEhVBNnvy4skcrXPnzQeY7UeV6bD0yWzYONe7Jv374KyjnZqszz6aefqmrSllTWLihDSDlbIzH3m7wTW9YXYyZP/6GSsHlIKrMP7ymnnFJxbZZlcCEfmIfcF3u9wcHBCrnpFf7mm29OlaEDX9XyS6ZuyqKiALAdwMUZyqIDQIdzvBPA30afnaF8gWupajlCFDsTO9fcuXNVVbW7u7tqA8+YMUNVK+N9lkql2LO1LeMjEuVFltKwXH7XqtLCRl1j8BwfGLIg9CC99NJLQeqz/c+PPPKIqqZtFTg6Y71UXLSktHRvF6yXZfm/eN158+bFeRjtDYYOnzVqYx6Odmx82oGBgVgZMMRAKCKcLwId5aUCWLp0aVAmhjIgfPFgQuB1WIbGdFl9z1oqP/nkk6qqunnz5oq8rMcG0mqIsgAwBcD7AMZEyuIggD4AmwCMi/KsB/Bjp8xGAAujz2NO+nUA1mdd77zzzqug5fIGheJ8uI2yZMmSYB4L1utG/coLny0FwZGKtYz0gR2VDwqjWflw2223qWryZrcdim91FxxGW9CYKw/yDPHJUGTeWnyRPP/886qaTzkzD8tk5fnkk09UNZmqMKyCa2FKhYeAMVgeWfIgNLJwwRGEHSlyyswyH3/8cXzOGp3Z61Gxu9i1a1f9lQWAUQD+AODvo+PvRlOLNgC/BLApSt/gURb/AGCRR1n8q+c6SwDsAbDHDR1olUUesAyNtPKAbwbejG3btnnfNqqqXV1duep0O8PGjRszZa2Wplq5FuC7Fr9DHcjNQ98S7e3tqpo8JPfdd1+cplo2JiM1mrLZYECHDx+O6/WF4asGTv9qgW0nTvlUk6kjafx2xEK41HGGAmRe2+esSUAe+joN4fIYemWBxn61wJop8B46Rm/1UxYAvh5NJ/4pcH4KgDe1ztMQd4GzHsY8BYYHxb0aGmp5EdJnRy3gGoZFPRc4BcAWAGtN+gTn93IAW6Pf3zcLnAeiEcjXot9TnQXO72dd21UWXJTjApi7aBYC83CBkJHMXXBYat/4XIl+8cUXVUS0VCoFV7Z5HYYtbG9vVxHR7du3a1tbm1500UWxDwH75ue36+zFPmxuOETVsvOblStXpiwzOdf31XH++ed7I3rxDcMpHRfr7r333op6+M03NX1Y2Do//PDDoAGWHQG4Ubq4A0QLSU6ziMsuuyyohKzPEPaJ3t7eYOQx+p9QVe3v7/fWa8sQbW1tQQO9PFOX0LqYajJdCuHiiy9OyeZGdwvJYdunra0tNkArlUr1MyQTkTkAfgfgDQBkmfwzgGsA/CAarh0EcKOqHonKrADwEwBfAPi5qv4mSl8AYG2kPDap6i+zrl2QsgoUaDzykrJqWuAc7o87smAsSV+g4jFjxqTiT9K5itW23LZzNXrIfRvMG6Kvr69ib3vdunXe67iwuxN0ZZe1LcfRDs9ZZzh09OMbtnL04/7HyZMn6/Tp0yuudfXVV6eOs3ZmCLttnGfKwYXDLJ5F6C1ufU264FZ5aJdn0qRJOnv27NQahl3Dcd3PWVns6CZrRGBlCE0pDh8+XOHn1Obt6OhQ1cSfpksbYN9gGa7x+NZNeMz6WMaNpl7XkUUzUYwshgetTE0u0HiMGLo3YWMu8Pjss8+ODZa2bNkCIDG8Ik2XsR9d3H333aljxpl0Y3G4OH78OF5//XUACS2X9TJew4033hjL1tHRket/ZRn7zJkzJ3XMBzrLyIq2I6S+09iKcMvecsstAID169en8qxZsyb+TXP8hQsXAkjiuBI+o7zQf7Km2AcOHMC6deu89XZ1dXnrcFEL1TkUY4RgNDkfbPsQ7v989tlnU+dsvN1q5V1cddVVVcuefvrpqWNGngPKLgCAtGm+C9LsrbFkVeQZfjTr46N7W5DhR5f49YJ165Znpdrd81ZNXO7XG9ddd13VPORU2K3NLFhafB4MxSHxokWLVLW21f8s2O1uH8j9sFO6eoPTTHfbvxq42MqQA42GJYphJLjVAxB3qFdffVVVK+fZQ0F7e3uwYyFaq7AdOktxWY/LLq9j+fLl2tbWVrF2QXKNTReReB5N79tEnoeLJLaHHnooLrN48WJ9+umnY3kIWz/BUAfd3d2xV+w8sKSfENnLF8uEWLNmjTfdxy2hO0CyeG+99daqMmbFbSHsrplViOSeWK/mLh588EFvHXwJPfzwwxVrHzfccEPqOrY8+R++PG1tbbmDErn/Z8WKFSNDWWQ9oNXsA1RVH3jggbhRVLWioUnb9dX7ZeB2IvqhZL18QPPIzyAxpFbnoT7zHDsOmYqEawNx5MgRVU28necZHdgOTqWaBVvvOeecE8zLhWxG8LriiiuqynbyySerquqpp54azEsP4HZBc9asWalj39ZlPcI5qCbGfgSVHPvlNddco6rJi9FXV0hZ1IIR67CXb0rSXzm8Y2NRG9Maz20MlmWEKg4RXZBjwCmEdbFO2GmJatjS0IW1IM0CrxlinubxJk4ZSMChC38XNr4KwXgcPoTYpKzjiy++iCOFVQPp2T09PSm+gwvr1t/XtqS6004ozzVpcckdBtsGPlDhW1invD5MmTIl3gUiR+jLgO1gpznPPPOMioiX48LRGs9ZS+QRoSymTZsW/0EGrrGNRnC4r5q8uaxZOS0lfbDko5C9R9b2IsuSWCQiOnv2bFVNyEz2zeC7uVQS3Oar5S3C7VzCegjv6uqqkIUKsrOzMziUr0WGLGqzC59nal7n/vvvV9Vk288F59xUEhzyZ21t2vpdD9chbNmyJfU/WMZOP9ygPTRQs0ZbPvaka6jnk5HTLN+5LISIf1wTseESR4SyyLPA2ShqcZ7pCOfk1cLdqSb8B2tXYB9uVY3XFxoNBqYZChiZLE/7c3ExC1Zpsv03bNiQW6bOzk5VTR7yLGRNhSz44vBxMohQO7DswMBAnEa/EtXK5kGekZG9jjWWHDHKwmpHgn4CAFRdrCTc9QI+tCzLtyGt/Dh/HAq44s4pkmoyt7fDbncdgsQaLkLa/075582bpzt27NATJ07E9YSsWvMs6NXytuLiZGho7vO1kQW3jXxlsoz1bJBjS6n3wY4sbMwQF/yP1R7Io0ePxtNY1uca4IXAUR+nJxyFkNLtgiNau76RBf5Xbgrw+LTTTtNjx47F+fIqi4KUVaBAHTBx4kQMDAw0VYbPPvssl1tIixFHyiJIZLHEpDxesumJ2dYFVPpxtHk+//zzirSnnnoKAHDGGWcASEhgpVIJU6dOTdVDr8+rVq1K1cG4D/Pnz4/9XJI0s3bt2lTexx57DEBCwJk+fXoFseemm24CUB4xunBlY6wU+qXkf37llVcQAmVbtGhRKt2Szz766KMgacwle4Vgy2aR1iwmT56cKuM6pSXo87SW+m0en19Q6+d15cqVAIBRo0YF6yWJj21rfX/u2bOnIr7K4OBgKs/Ro0cBlPu/VRT0X0uSF0mFQ0UxsihQ4CuOETOyYGQnGzekp6cHQPmtSGozEYr+xbegiAQ9KNt4Ei74trBRxiwWLFhQkVZL9KcQDTfLy/Rzzz0HIHnj2JGNT5ZHH300lc5YI7637dixYzPrCp13wXu5c+fOinOMnUGEYpoAiSdul+IMhEed7jkLxp/xlWF72HOM9uYi5F2dUcB812ca62O8k82bN2fK7KuDOHToUDAGC2O0DBXFyKIBGK5YnqVSKR6WFkiwd+/eeNqXF2eddVZFeL+vCkbMyMIiZNQDAPv37weQrE0cPHgQQKJ9x40bV1Gmr68PQKUh0a5duwAA3d3dFWWoqV977TUAlXEYXEVxzz33pMrw7Xr99dcDSGJkuli6dCkAYPny5QAqo04xfXBwENu2bQNQaciUBbahfRsyjogPtcTUDOVhjJEnnngiTmNcFeKtt94CkBjE8R664P3k94UXXpg6v3r1agDAmWeeiXPPPRdAcq+sdS3r8C1OunKGZKkGO2JyccEFFwBI7ocv/muov/f29gJI+sb48ePjcx988AGAZG0lNAK2RoZVkWfLpFkfl2dhqbIkXGVtk40ePTp1nIc7wa2voex9czuO/jR84JYtvTnlkYm2Gnfeeaeq5uMR1AIfO7UaLOcgT3tZirv1Mu3CkuL6+vqCeck8RbStSNJZFkIhAeiP1AW9w3PbmMS1kJs61YRKz+1u11V/NSxbtix1/PLLL1fkoYEgKeN5+pH1rk5gpPAsrIduX6ckE45BW+zeOZ2vuIw5GzuDnrSz7C9sGm9YiAuimlDC2ZHJwSCpafXq1UGryTz2I+SbkPGYx34ktEfPsiQ3uWlUcrajlUqlCko77TBsu7gOf9h2tM9hHipaOojJ+h98GeRx2kNk1UfjO/JcrN2IL7YIQfnZx6688sqqeUnZpqkC09k3fPLaPkLa+okTJ3w2H6k6XDshhnuI8o0MZVEPkDb9wgsvqGpiH8DOlvVmpY2CD+xMlmufxzyZD7l9+HwIKY0xY8bE7FFG9LKK0nYuV2HSSIsK7f333/eW8YHxVmj4ddddd6XOu5a4Vlkw/okvWhrLMRBO6OEWkZhWT3o8vai5D1s1UDZrJerD1q1bc9fLBxWG3j927NgKv7E8N2HChFS6jWnjwo68fPeMeXyWzaqJ79G8yqKlFzhF5C8ohztsRXwHwIfNFiKAQrah4asq21+p6qnVMvlXPloH+zWPI9EmQET2FLLVjkK2oaEVZPt/txtSoECB5qBQFgUKFMiFVlcW/9ZsATJQyDY0FLINDU2XraUXOAsUKNA6aPWRRYECBVoEhbIoUKBALrSsshCRS0Vkv4j0i8jtTZLhoIi8ISK9IrInShsvIp0i8m70PS5KFxFZF8nbJyIz6yzLJhE5LiJvOmk1yyIii6P874rI4gbJdYeIDETt1ivlGLc81xHJtV9E5jvpdb/fIjJJRHaJyNsisk9Efhalt0K7hWRribbzIg9za7g/KAdOfg/ANCQR12c0QY6DAL5j0u4HcHv0+3YA90W/FwD4DcpR52cB2F1nWeYCmAngzaHKAmA8ypHsxwMYF/0e1wC57gDwC0/eGdG9PAnA1Ogelxp1vwFMADAz+j0awDuRDK3QbiHZWqLtfJ9WHVn8DYB+VT2gqp8D2Arg8ibLRFwOYHP0ezOAK5x0Wnj9HsC3RGRCvS6qqt0A/vwlZZkPoFNV/6yqHwPoBHBpA+QK4XIAW1X1f1T1jwD6Ub7XDbnfqnpEVV+Pfv8FwNsAJqI12i0kWwjD2nY+tKqymAjgv5zjPyG7IRsFBfBbEfmDiCyJ0r6rqkeA8g0HcFqU3gyZa5VlOGW8ORrKb+Iwv5lyicgUAOcC2I0WazcjG9BibUe0qrLwubFqxh7v36nqTAA/BPBTEZmbkbdVZAbCsgyXjI8A+GsAPwBwBMCDzZRLREYBeB7Az1X1v7OyBuRomHwe2Vqq7Vy0qrL4E4BJzvEZAKqHpa4zVPVw9H0cwAsoD/mOcXoRfdPzTTNkrlWWYZFRVY+p6qCqngDw7yi3W1PkEpGvo/ww/oeq/jpKbol288nWSm1n0arK4j8BfE9EporINwD8CMCO4RRARL4pIqP5G8AlAN6M5OBq+GIA26PfOwBcH62ozwLwKYe6DUStsuwEcImIjIuGt5dEaXWFWau5EuV2o1w/EpGTRGQqgO8BeA0Nut8iIgA2AnhbVf/FOdX0dgvJ1ipt50UjVk3r8UF5ZfodlFd6VzTh+tNQXlneC2AfZQDwbQAvAXg3+h4fpQuADZG8bwA4v87yPIXysPR/UX6b/ONQZAHwE5QXx/oB3NAguX4VXbcP5Y47wcm/IpJrP4AfNvJ+A5iD8pC8D0Bv9FnQIu0Wkq0l2s73KejeBQoUyIVWnYYUKFCgxVAoiwIFCuRCoSwKFCiQC4WyKFCgQC4UyqJAgQK5UCiLAgUK5EKhLAoUKJAL/weopXDcBpkKkgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "import os\n",
    "from tensorflow.examples.tutorials.mnist import input_data\n",
    "\n",
    "# PROJECTOR需要的日志文件名和地址相关参数\n",
    "LOG_DIR = 'log/high_dim'\n",
    "SPRITE_FILE = 'mnist_sprite.jpg'\n",
    "META_FIEL = \"mnist_meta.tsv\"\n",
    "\n",
    "# 使用给出的MNIST图片列表生成sprite图像\n",
    "def create_sprite_image(images):\n",
    "    \"\"\"Returns a sprite image consisting of images passed as argument. Images should be count x width x height\"\"\"\n",
    "    if isinstance(images, list):\n",
    "        images = np.array(images)\n",
    "    img_h = images.shape[1]\n",
    "    img_w = images.shape[2]\n",
    "    # sprite图像可以理解成是所有小图片拼成的大正方形矩阵，大正方形矩阵中的每一个\n",
    "    # 元素就是原来的小图片。于是这个正方形的地长就是sqrt(n)，其中n为小图片的数量。\n",
    "    n_plots = int(np.ceil(np.sqrt(images.shape[0])))\n",
    "    \n",
    "    # 使用全1来初始化最终的大图片\n",
    "    spriteimage = np.ones((img_h * n_plots ,img_w * n_plots ))\n",
    "    \n",
    "    for i in range(n_plots):\n",
    "        for j in range(n_plots):\n",
    "            # 计算当前图片的编号\n",
    "            this_filter = i * n_plots + j\n",
    "            if this_filter < images.shape[0]:\n",
    "                # 将当前小图片的内容复制到最终的sprite图像\n",
    "                this_img = images[this_filter]\n",
    "                spriteimage[i * img_h:(i + 1) * img_h,\n",
    "                  j * img_w:(j + 1) * img_w] = this_img\n",
    "    \n",
    "    return spriteimage\n",
    "\n",
    "# 加载MNIST数据，这列onehot为False，得到的labels为数字\n",
    "mnist = input_data.read_data_sets(\"../../datasets/MNIST_data\", one_hot=False)\n",
    "\n",
    "# 生成sprite图像\n",
    "to_visualise = 1 - np.reshape(mnist.test.images,(-1,28,28))\n",
    "sprite_image = create_sprite_image(to_visualise)\n",
    "\n",
    "# 将生成的sprite图像放到相应的日志目录下\n",
    "path_for_mnist_sprites = os.path.join(LOG_DIR, SPRITE_FILE)\n",
    "plt.imsave(path_for_mnist_sprites,sprite_image,cmap='gray')\n",
    "plt.imshow(sprite_image,cmap='gray')\n",
    "\n",
    "# 生成每张图片对应的标签文件并写到相应的日志目录下。\n",
    "path_for_mnist_metadata = os.path.join(LOG_DIR, META_FIEL)\n",
    "with open(path_for_mnist_metadata,'w') as f:\n",
    "    f.write(\"Index\\tLabel\\n\")\n",
    "    for index,label in enumerate(mnist.test.labels):\n",
    "        f.write(\"%d\\t%d\\n\" % (index,label))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "运行以上代码可以得到两个文件， 一个是如图11.22所示的MNIST测试数据sprite图像，这个图像包含了所有的MNIST测试数据图像。另一个是mnist_meta.tsv，下面给出了这个tsv文件的前面几行。可以看出，这个文件的第一行是每一列的说明，以后的每一行代表一张图片，在这个文件中给出了每一张图对应的真实标签。\n",
    "\n",
    "| Index| Label |\n",
    "|:----:|:-----:|\n",
    "|   0 |   7  |\n",
    "|   1 |   2  |\n",
    "|   2 |   1  |\n",
    "|   3 |   0  |\n",
    "|   4 |   4  |\n",
    "\n",
    "<p align='center'>\n",
    "    <img src=images/mnist_sprite.jpg>\n",
    "    <center>图11-22 使用MNIST测试数据集生成的sprite图像\n",
    "</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在生成好辅助数据之后，以下代码展示了如何使用TensorFlow代码生成PROJECTOR所需要的日志文件来可视化MNIST测试数据在最后的输出层向量。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting ../../datasets/MNIST_data\\train-images-idx3-ubyte.gz\n",
      "Extracting ../../datasets/MNIST_data\\train-labels-idx1-ubyte.gz\n",
      "WARNING:tensorflow:From d:\\python3\\tfgpu\\dl+\\lib\\site-packages\\tensorflow\\contrib\\learn\\python\\learn\\datasets\\mnist.py:110: dense_to_one_hot (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use tf.one_hot on tensors.\n",
      "Extracting ../../datasets/MNIST_data\\t10k-images-idx3-ubyte.gz\n",
      "Extracting ../../datasets/MNIST_data\\t10k-labels-idx1-ubyte.gz\n",
      "After 0 training step(s), loss on training batch is 2.99054.\n",
      "After 1000 training step(s), loss on training batch is 0.183446.\n",
      "After 2000 training step(s), loss on training batch is 0.146804.\n",
      "After 3000 training step(s), loss on training batch is 0.14023.\n",
      "After 4000 training step(s), loss on training batch is 0.116306.\n",
      "After 5000 training step(s), loss on training batch is 0.104121.\n",
      "After 6000 training step(s), loss on training batch is 0.0938725.\n",
      "After 7000 training step(s), loss on training batch is 0.0926095.\n",
      "After 8000 training step(s), loss on training batch is 0.0888463.\n",
      "After 9000 training step(s), loss on training batch is 0.0814582.\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "import mnist_inference\n",
    "import os\n",
    "\n",
    "# 加载用于生成PROJECTOR日志的帮助函数。\n",
    "from tensorflow.contrib.tensorboard.plugins import projector\n",
    "from tensorflow.examples.tutorials.mnist import input_data\n",
    "\n",
    "# 和第5章中类似地定义训练模型需要的参数。这里我们同样是复用第5章中定义的\n",
    "# mnist_inference过程。\n",
    "BATCH_SIZE = 100\n",
    "LEARNING_RATE_BASE = 0.8\n",
    "LEARNING_RATE_DECAY = 0.99\n",
    "REGULARIZATION_RATE = 0.0001\n",
    "TRAINING_STEPS = 10000\n",
    "MOVING_AVERAGE_DECAY = 0.99\n",
    "\n",
    "LOG_DIR = 'log/high_dim'\n",
    "SPRITE_FILE = 'mnist_sprite.jpg'\n",
    "META_FIEL = \"mnist_meta.tsv\"\n",
    "TENSOR_NAME = \"FINAL_LOGITS\"\n",
    "\n",
    "# 训练过程和第5章给出来的基本一致，唯一不同的是这里还需要返回最后测试数据经过整个\n",
    "# 神经网络得到的输出层矩阵（因为有多张测试图片，每张图片对应了一个输出层向量，所以\n",
    "# 返回的结果是这些向量组成的矩阵〉。\n",
    "def train(mnist):\n",
    "    #  输入数据的命名空间。\n",
    "    with tf.name_scope('input'):\n",
    "        x = tf.placeholder(tf.float32, [None, mnist_inference.INPUT_NODE], name='x-input')\n",
    "        y_ = tf.placeholder(tf.float32, [None, mnist_inference.OUTPUT_NODE], name='y-input')\n",
    "    regularizer = tf.contrib.layers.l2_regularizer(REGULARIZATION_RATE)\n",
    "    y = mnist_inference.inference(x, regularizer)\n",
    "    global_step = tf.Variable(0, trainable=False)\n",
    "    \n",
    "    # 处理滑动平均的命名空间。\n",
    "    with tf.name_scope(\"moving_average\"):\n",
    "        variable_averages = tf.train.ExponentialMovingAverage(MOVING_AVERAGE_DECAY, global_step)\n",
    "        variables_averages_op = variable_averages.apply(tf.trainable_variables())\n",
    "   \n",
    "    # 计算损失函数的命名空间。\n",
    "    with tf.name_scope(\"loss_function\"):\n",
    "        cross_entropy = tf.nn.sparse_softmax_cross_entropy_with_logits(logits=y, labels=tf.argmax(y_, 1))\n",
    "        cross_entropy_mean = tf.reduce_mean(cross_entropy)\n",
    "        loss = cross_entropy_mean + tf.add_n(tf.get_collection('losses'))\n",
    "    \n",
    "    # 定义学习率、优化方法及每一轮执行训练的操作的命名空间。\n",
    "    with tf.name_scope(\"train_step\"):\n",
    "        learning_rate = tf.train.exponential_decay(\n",
    "            LEARNING_RATE_BASE,\n",
    "            global_step,\n",
    "            mnist.train.num_examples / BATCH_SIZE, LEARNING_RATE_DECAY,\n",
    "            staircase=True)\n",
    "\n",
    "        train_step = tf.train.GradientDescentOptimizer(learning_rate).minimize(loss, global_step=global_step)\n",
    "\n",
    "        with tf.control_dependencies([train_step, variables_averages_op]):\n",
    "            train_op = tf.no_op(name='train')\n",
    "    \n",
    "    # 训练模型。\n",
    "    with tf.Session() as sess:\n",
    "        tf.global_variables_initializer().run()\n",
    "        for i in range(TRAINING_STEPS):\n",
    "            xs, ys = mnist.train.next_batch(BATCH_SIZE)\n",
    "            _, loss_value, step = sess.run([train_op, loss, global_step], feed_dict={x: xs, y_: ys})\n",
    "                \n",
    "            if i % 1000 == 0:\n",
    "                print(\"After %d training step(s), loss on training batch is %g.\" % (i, loss_value))\n",
    "        \n",
    "        # 计算MNIST测试数据对应的输出层矩阵。\n",
    "        final_result = sess.run(y, feed_dict={x: mnist.test.images})\n",
    "    \n",
    "    # 返回输出层矩阵的值\n",
    "    return final_result\n",
    "\n",
    "\n",
    "# 生成可视化最终输出层向量所需要的日志文件。\n",
    "def visualisation(final_result):\n",
    "    # 使用一个新的变量来保存最终输出层向量的结果。因为embedding是通过TensorFlow\n",
    "    # 中变量完成的，所以PROJECTOR可视化的都是TensorFlow中的变量。于是这里要\n",
    "    # 新定义一个变量来保存输出层向量的取值。\n",
    "    y = tf.Variable(final_result, name=TENSOR_NAME)\n",
    "    summary_writer = tf.summary.FileWriter(LOG_DIR)\n",
    "\n",
    "    # 通过projector.ProjectorConfig类来帮助生成日志文件。\n",
    "    config = projector.ProjectorConfig()\n",
    "    # 增加一个需要可视化的embedding结果\n",
    "    embedding = config.embeddings.add()\n",
    "    # 指定这个embedding结果对应的TensorFlow变量名称\n",
    "    embedding.tensor_name = y.name\n",
    "\n",
    "    # 指定embedding结果所对应的原始数据信息。比如这里指定的就是每一张MNIST测试\n",
    "    # 图片对应的真实类别。在单词向量中可以是单词ID对应的单词。这个文件是可选的，\n",
    "    # 如果没有指定那么向量就没有标签。\n",
    "    embedding.metadata_path = META_FIEL\n",
    "\n",
    "    # 指定sprite图像。这个也是可选的，如果没有提供sprite图像，那么可视化的结果\n",
    "    # 每一个点就是一个小圆点，而不是具体的图片。\n",
    "    embedding.sprite.image_path = SPRITE_FILE\n",
    "    # 在提供sprite图像时，通过single_image_dim可以指定单张图片的大小。\n",
    "    # 这将用于从sprite图像中截取正确的原始图片。\n",
    "    embedding.sprite.single_image_dim.extend([28,28])\n",
    "\n",
    "    # 将PROJECTOR所需要的内容写入日志文件。\n",
    "    projector.visualize_embeddings(summary_writer, config)\n",
    "    \n",
    "    # 生成会话，初始化新声明的变量并将需要的日志信息写入文件。\n",
    "    sess = tf.InteractiveSession()\n",
    "    sess.run(tf.global_variables_initializer())\n",
    "    saver = tf.train.Saver()\n",
    "    saver.save(sess, os.path.join(LOG_DIR, \"model\"), TRAINING_STEPS)\n",
    "    \n",
    "    summary_writer.close()\n",
    "    \n",
    "\n",
    "# 主函数\n",
    "def main(argv=None): \n",
    "    mnist = input_data.read_data_sets(\"../../datasets/MNIST_data\", one_hot=True)\n",
    "    final_result = train(mnist)\n",
    "    visualisation(final_result)\n",
    "\n",
    "if __name__ == '__main__':\n",
    "    main()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*注意这一部分书上版本略有落后，可参考[projector官网](https://projector.tensorflow.org)，另外小图片一直操作不出来，待改进*\n",
    "\n",
    "运行以上代码就可以得到PROJECTOR所需要的所有日志文件。在LOGDIR下启动TensorBoard就可以看到类似图11.23所示的效果图。从图11.23(a)中可以看出，在17轮迭代之后，模型的分类效果不是很理想，所以图11.23(a)上不同颜色的图片（代表不同的类别）混乱地挤在一起。但是当迭代100轮之后从图11.23(b)上可以明显地看出不同颜色的图片的区分度还是比较大的。\n",
    "<p align='center'>\n",
    "    <img src=images/图11.23a.JPG>\n",
    "    <center>(a) 使用5.5节给出的网络结构在训练17轮之后的结果</center>\n",
    "    <img src=images/图11.23b.JPG>\n",
    "    <center>(b) 使用5.5节给出的网络结构在训练100轮之后的结果</center>\n",
    "    <center>图11-23 使用T-SNE可视化得到的MNIST测试数据在最后输出层的向量</center>\n",
    "</p>\n",
    "\n",
    "在**PROJECTOR界面的左上角是数据面板，**有若干个选项：\n",
    "- 第一个“FINAL_LOGITS”选项是选择需要可视化的Tensor，这里默认选择的是通过ProjectorConfig中指定的tensor_name，也就是名为FINAL_LOGITS的张量。点开这个选项可以看到其他可以可视化结果，虽然PROJECTOR也可以可视化这些向量，但是在这个场景下意义不大；\n",
    "- 第二个“Label by”选项可以控制当鼠标移到一个向量上时鼠标附近显示的标签。比如这里选定的是“Index”，那么当鼠标移到某个点上之后显示的就是这个点对应的编号。\n",
    "- 第三个“Color by”选项可以指定每一个小图片的背景颜色。图11.23中小图片的背景颜色都是根据标签选取的，所以拥有相同标签的样例的背景颜色是一样的。这样可以快速区分哪些样例属于同一个种类。\n",
    "- 第四个是“Suprevised with”，最后一个是“Edit by”，这两个功能还有待摸索。\n",
    "\n",
    "在**PROJECTOR界面的左下角是投影面板，**提供了不同的高维向量的可视化方法，目前主要支持的就是T-SNE和PCA。无论是T-SNE还是PCA都可以将一个高维向量转化成一个低维向量，并尽量保证转化后向量中的信息不受影响。因为篇幅的关系本书中不再详细介绍这些具体的算法。\n",
    "\n",
    "在**PROJECTOR界面的右上角是检查工具面板，**提供了高亮功能。图展示了搜索所有代表数字3的图片，可以看出所有代表数字3的图片都被高亮标出来了，而且大部分的图片都集中在一个比较小的区域，只有少数离中心区域比较远。通过这种方式可以很快地找到每个类别中比较难分的图片，加速错误案例分析的过程。\n",
    "<p align='center'>\n",
    "    <img src=images/图11.26.JPG>\n",
    "    <center>图11-26 PROJECTOR界面的搜索高亮功能</center>\n",
    "</p>\n",
    "\n",
    "在**PROJECTOR界面的右下角**提供了分享功能，可以将当前状态保存为小文件的功能，接着可以将PROJECTOR指向一个包含一个或多个这些文件的集合，方便其他用户查看一系列书签。"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
